[jira] [Updated] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-3755: Attachment: SOLR-3755.patch Changes: # Fixed off-by-one mistake in replica creation # Commit before split # Better sub-shard node addition logic using shard ranges. Defensive checks in DUPF are modified to allow sub-shard replication. A new param distrib.from.shard is added to forwarded requests if node list contains sub-shard leaders. # Modified ShardSplitTest to index docs constantly (to check sub-shard replication). The test fails currently because the sub-shards now have more docs than they should have. I'm investigating this. Patch is created on top of trunk r1459441 via git. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-3755: Attachment: SOLR-3755.patch Changes: # Make splitshard retryable by unloading cores part of a sub-shard and starting from scratch again. If a sub-shard is already in active state, splitshard will fail. # Use PRERECOVERY core admin command to make the parent shard leader wait for sub shard cores to be active # Similar to point 2 above, use the PRERECOVERY command to wait for sub shard replicas to be available before switching the shard states # The mismatch between the expected and actual number of documents was because the SolrIndexSplitter does not take the type of the unique key field into account while calculating the hash. I changed it to use indexedToReadable to make it compute the correct hash. This needs to be reviewed for performance implications. # ShardSplitTest passes. I'll be working to add more tests in the coming days. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-1741) NPE when deletionPolicy sets maxOptimizedCommitsTokeep=0
[ https://issues.apache.org/jira/browse/SOLR-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-1741. - Resolution: Cannot Reproduce > NPE when deletionPolicy sets maxOptimizedCommitsTokeep=0 > > > Key: SOLR-1741 > URL: https://issues.apache.org/jira/browse/SOLR-1741 > Project: Solr > Issue Type: Bug > Components: replication (java) >Affects Versions: 1.4 >Reporter: Noble Paul >Assignee: Shalin Shekhar Mangar >Priority: Minor > Fix For: 4.3 > > > This is a user reported issue http://markmail.org/thread/bjcwiw3s66b5x76h -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13618065#comment-13618065 ] Shalin Shekhar Mangar commented on SOLR-3755: - Okay, the test still fails sometimes. I'm looking into it. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-2775) DataImportHandler jdbc password
[ https://issues.apache.org/jira/browse/SOLR-2775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-2775. - Resolution: Won't Fix Variables specified in data config are also resolved with system properties so there is no reason anymore to configure DIH via solrconfig. Hence, the bug can be worked around by using dataconfig instead of configuring via solrconfig. > DataImportHandler jdbc password > --- > > Key: SOLR-2775 > URL: https://issues.apache.org/jira/browse/SOLR-2775 > Project: Solr > Issue Type: Bug > Components: contrib - DataImportHandler >Affects Versions: 4.0-ALPHA > Environment: All >Reporter: Des Lownds >Assignee: Shalin Shekhar Mangar >Priority: Minor > Fix For: 4.3 > > > The http response from dataimporthandler, as well as logging output exposes > the jdbc password in plain text. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13621060#comment-13621060 ] Shalin Shekhar Mangar commented on SOLR-3755: - I'd like to commit the patch to 4x and trunk soon. We can then work on improving the features and tests via the regular route. If there are no objections, I'll commit it tomorrow. bq. It's very common for these types of tests to be sensitive to the exact env (hardware, OS, etc). A lot of times it's some timing issue. Yeah, I'm still trying to reproduce the issue. I'll try to find a solution before I commit. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13622208#comment-13622208 ] Shalin Shekhar Mangar commented on SOLR-3755: - I ran into another bug. Adding mutable state in cloud descriptor (like shard state and range) is a bad idea. The sub shard cores are created while the sub shard is in construction state therefore their cloud descriptor keeps "construction" as the shard state. If the sub shard leader goes down after the shard state has been changed to "active", it sets the shard state to "construction" once again while publishing itself as "down". > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13623541#comment-13623541 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. The sub shard cores are created while the sub shard is in construction state therefore their cloud descriptor keeps "construction" as the shard state. If the sub shard leader goes down after the shard state has been changed to "active", it sets the shard state to "construction" once again while publishing itself as "down". I've fixed it in the git branch. Although I don't like the fix very much. In the git branch, I'm using the shardState and shardRange fields in CloudDescriptor for a one-time usage. They are set to null once the new sub shard core is registered (and the new sub shard is created in zk). Maybe shardState and shardRange should be a core property instead? > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4682) CoreAdminRequest.mergeIndexes can not merge mutilple cores.
[
https://issues.apache.org/jira/browse/SOLR-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624346#comment-13624346
]
Shalin Shekhar Mangar commented on SOLR-4682:
-
Jason, where is the code that you have put in the issue description? I see the
following in CoreAdminRequest on branch 4.2:
{code}
public static CoreAdminResponse mergeIndexes(String name,
String[] indexDirs, String[] srcCores, SolrServer server) throws
SolrServerException,
IOException {
CoreAdminRequest.MergeIndexes req = new CoreAdminRequest.MergeIndexes();
req.setCoreName(name);
req.setIndexDirs(Arrays.asList(indexDirs));
req.setSrcCores(Arrays.asList(srcCores));
return req.process(server);
}
{code}
> CoreAdminRequest.mergeIndexes can not merge mutilple cores.
> ---
>
> Key: SOLR-4682
> URL: https://issues.apache.org/jira/browse/SOLR-4682
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 4.2
> Environment: java version "1.6.0_20"
>Reporter: Jason.D.Cao
> Labels: patch
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The mergeIndexes method in CoreAdminRequest class accepts an array of
> srcCores, but it only handles the last element in srcCores array. Related
> code as follows,
> if (srcCores != null) {
> for (String srcCore : srcCores) {
> params.set(CoreAdminParams.SRC_CORE, srcCore);
> }
> }
> The for-each loop above just override the SRC_CORE value in params and only
> the last one reserved finally.
> We should remove the for-each loop and set SRC_CORE value with an array of
> srcCores.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4682) CoreAdminRequest.mergeIndexes can not merge mutilple cores.
[
https://issues.apache.org/jira/browse/SOLR-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13624347#comment-13624347
]
Shalin Shekhar Mangar commented on SOLR-4682:
-
Ok, I see that code is in the class CoreAdminRequest.MergeIndexes. I'll fix.
> CoreAdminRequest.mergeIndexes can not merge mutilple cores.
> ---
>
> Key: SOLR-4682
> URL: https://issues.apache.org/jira/browse/SOLR-4682
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 4.2
> Environment: java version "1.6.0_20"
>Reporter: Jason.D.Cao
> Labels: patch
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The mergeIndexes method in CoreAdminRequest class accepts an array of
> srcCores, but it only merge the last core of array into targetCore. Related
> code as follows,
> if (srcCores != null) {
> for (String srcCore : srcCores) {
> params.set(CoreAdminParams.SRC_CORE, srcCore);
> }
> }
> The for-each loop above overrides the SRC_CORE value in params and only the
> last one reserved when loop ends.
> We should remove the for-each loop and set SRC_CORE value with an array of
> srcCores.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4682) CoreAdminRequest.mergeIndexes can not merge mutilple cores.
[
https://issues.apache.org/jira/browse/SOLR-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-4682:
Attachment: SOLR-4682.patch
Fix with a test.
> CoreAdminRequest.mergeIndexes can not merge mutilple cores.
> ---
>
> Key: SOLR-4682
> URL: https://issues.apache.org/jira/browse/SOLR-4682
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 4.2
> Environment: java version "1.6.0_20"
>Reporter: Jason.D.Cao
> Labels: patch
> Attachments: SOLR-4682.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The mergeIndexes method in CoreAdminRequest class accepts an array of
> srcCores, but it only merge the last core of array into targetCore. Related
> code as follows,
> if (srcCores != null) {
> for (String srcCore : srcCores) {
> params.set(CoreAdminParams.SRC_CORE, srcCore);
> }
> }
> The for-each loop above overrides the SRC_CORE value in params and only the
> last one reserved when loop ends.
> We should remove the for-each loop and set SRC_CORE value with an array of
> srcCores.
> The code above is in CoreAdminRequest class line 330
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4682) CoreAdminRequest.mergeIndexes can not merge mutilple cores.
[
https://issues.apache.org/jira/browse/SOLR-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar resolved SOLR-4682.
-
Resolution: Fixed
Fix Version/s: 5.0
4.3
Assignee: Shalin Shekhar Mangar
Fixed in trunk and branch_4x.
> CoreAdminRequest.mergeIndexes can not merge mutilple cores.
> ---
>
> Key: SOLR-4682
> URL: https://issues.apache.org/jira/browse/SOLR-4682
> Project: Solr
> Issue Type: Bug
> Components: clients - java
>Affects Versions: 4.2
> Environment: java version "1.6.0_20"
>Reporter: Jason.D.Cao
>Assignee: Shalin Shekhar Mangar
> Labels: patch
> Fix For: 4.3, 5.0
>
> Attachments: SOLR-4682.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The mergeIndexes method in CoreAdminRequest class accepts an array of
> srcCores, but it only merge the last core of array into targetCore. Related
> code as follows,
> if (srcCores != null) {
> for (String srcCore : srcCores) {
> params.set(CoreAdminParams.SRC_CORE, srcCore);
> }
> }
> The for-each loop above overrides the SRC_CORE value in params and only the
> last one reserved when loop ends.
> We should remove the for-each loop and set SRC_CORE value with an array of
> srcCores.
> The code above is in CoreAdminRequest class line 330
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-3755: Attachment: SOLR-3755.patch Patch updated to trunk. Changes: # Router and range for the current core is used during split # Change sub-shard replication to check only sub shard range instead of state # Fixed replica allocation code such that replicas are not created on the same node always # Splitting a sub-shard works # Slice state and range kept in CloudDescriptor is used one-time only. DUPF and other places rely on state/range read from ZK. # Removed multiple debug statements, sleeps and nocommits > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755-testSplitter.patch, SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-3755: Attachment: SOLR-3755.patch Adding patch that I've committed to trunk and branch_4x. Changes: 1. Changed param name from "name" to "collection" for splitshard api 2. Added a comment warning not to use shardState and shardRange from CloudDescriptor > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4655) The Overseer should assign node names by default.
[ https://issues.apache.org/jira/browse/SOLR-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1362#comment-1362 ] Shalin Shekhar Mangar commented on SOLR-4655: - Mark, I noticed this issue after I committed SOLR-3755. We assign names to sub-shard nodes in OverseerCollectionProcessor. Is the change as simple as using Assign.assignNode() or is there something more I need to take care of? > The Overseer should assign node names by default. > - > > Key: SOLR-4655 > URL: https://issues.apache.org/jira/browse/SOLR-4655 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 4.3, 5.0 > > Attachments: SOLR-4655.patch > > > Currently we make a unique node name by using the host address as part of the > name. This means that if you want a node with a new address to take over, the > node name is misleading. It's best if you set custom names for each node > before starting your cluster. This is cumbersome though, and cannot currently > be done with the collections API. Instead, the overseer could assign a more > generic name such as nodeN by default. Then you can easily swap in another > node with no pre planning and no confusion in the name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-3755: Fix Version/s: 5.0 4.3 Assignee: Shalin Shekhar Mangar > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 4.3, 5.0 > > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4655) The Overseer should assign node names by default.
[ https://issues.apache.org/jira/browse/SOLR-4655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13625718#comment-13625718 ] Shalin Shekhar Mangar commented on SOLR-4655: - Sub-slice names are created with just appending _N to the parent shard name. For example, shard1 gets split into shard1_0 and shard1_1 etc. The node names are created as collection_shard1_0_replica1, collection_shard1_0_replica2 etc. > The Overseer should assign node names by default. > - > > Key: SOLR-4655 > URL: https://issues.apache.org/jira/browse/SOLR-4655 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 4.3, 5.0 > > Attachments: SOLR-4655.patch, SOLR-4655.patch > > > Currently we make a unique node name by using the host address as part of the > name. This means that if you want a node with a new address to take over, the > node name is misleading. It's best if you set custom names for each node > before starting your cluster. This is cumbersome though, and cannot currently > be done with the collections API. Instead, the overseer could assign a more > generic name such as nodeN by default. Then you can easily swap in another > node with no pre planning and no confusion in the name. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4530) DIH: Provide configuration to use Tika's IdentityHtmlMapper
[ https://issues.apache.org/jira/browse/SOLR-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-4530. - Resolution: Fixed Assignee: Shalin Shekhar Mangar Committed to trunk and branch_4x > DIH: Provide configuration to use Tika's IdentityHtmlMapper > --- > > Key: SOLR-4530 > URL: https://issues.apache.org/jira/browse/SOLR-4530 > Project: Solr > Issue Type: Improvement > Components: contrib - DataImportHandler >Affects Versions: 4.1 >Reporter: Alexandre Rafalovitch >Assignee: Shalin Shekhar Mangar >Priority: Minor > Fix For: 4.3 > > Attachments: SOLR-4530.patch > > > When using TikaEntityProcessor in DIH, the default HTML Mapper strips out > most of the HTML. It may make sense when the expectation is just to store the > extracted content as a text blob, but DIH allows more fine-tuned content > extraction (e.g. with nested XPathEntityProcessor). > Recent Tika versions allow to set an alternative HTML Mapper implementation > that passes all the HTML in. It would be useful to be able to set that > implementation from DIH configuration. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4581) sort-order of facet-counts depends on facet.mincount
[
https://issues.apache.org/jira/browse/SOLR-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-4581:
Attachment: SOLR-4581.patch
Test to demonstrate the issue.
> sort-order of facet-counts depends on facet.mincount
>
>
> Key: SOLR-4581
> URL: https://issues.apache.org/jira/browse/SOLR-4581
> Project: Solr
> Issue Type: Bug
>Affects Versions: 4.2
>Reporter: Alexander Buhr
> Attachments: SOLR-4581.patch
>
>
> I just upgraded to Solr 4.2 and cannot explain the following behaviour:
> I am using a solr.TrieDoubleField named 'ListPrice_EUR_INV' as a facet-field.
> The solr-response for the query
> {noformat}'solr/Products/select?q=*%3A*&wt=xml&indent=true&facet=true&facet.field=ListPrice_EUR_INV&f.ListPrice_EUR_INV.facet.sort=index'{noformat}
> includes the following facet-counts:
> {noformat}
> 1
> 1
> 1
> {noformat}
> If I also set the parameter *'facet.mincount=1'* in the query, the order of
> the facet-counts is reversed.
> {noformat}
> 1
> 1
> 1
> {noformat}
> I would have expected, that the sort-order of the facet-counts is not
> affected by the facet.mincount parameter, as it is in Solr 4.1.
> Is this related to SOLR-2850?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4581) sort-order of facet-counts depends on facet.mincount
[
https://issues.apache.org/jira/browse/SOLR-4581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-4581:
Attachment: SOLR-4581.patch
The bug is reproducible even after reverting SOLR-2850.
Adding facet.method=fc gives correct response but omitting facet.method or
using facet.method=enum gives the wrong sort order.
I'm not that familiar with faceting code to fix this. Perhaps someone else can
take a look.
> sort-order of facet-counts depends on facet.mincount
>
>
> Key: SOLR-4581
> URL: https://issues.apache.org/jira/browse/SOLR-4581
> Project: Solr
> Issue Type: Bug
>Affects Versions: 4.2
>Reporter: Alexander Buhr
> Attachments: SOLR-4581.patch, SOLR-4581.patch
>
>
> I just upgraded to Solr 4.2 and cannot explain the following behaviour:
> I am using a solr.TrieDoubleField named 'ListPrice_EUR_INV' as a facet-field.
> The solr-response for the query
> {noformat}'solr/Products/select?q=*%3A*&wt=xml&indent=true&facet=true&facet.field=ListPrice_EUR_INV&f.ListPrice_EUR_INV.facet.sort=index'{noformat}
> includes the following facet-counts:
> {noformat}
> 1
> 1
> 1
> {noformat}
> If I also set the parameter *'facet.mincount=1'* in the query, the order of
> the facet-counts is reversed.
> {noformat}
> 1
> 1
> 1
> {noformat}
> I would have expected, that the sort-order of the facet-counts is not
> affected by the facet.mincount parameter, as it is in Solr 4.1.
> Is this related to SOLR-2850?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-4695) Fix core admin SPLIT action to be useful with non-cloud setups
Shalin Shekhar Mangar created SOLR-4695: --- Summary: Fix core admin SPLIT action to be useful with non-cloud setups Key: SOLR-4695 URL: https://issues.apache.org/jira/browse/SOLR-4695 Project: Solr Issue Type: Bug Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: 4.3 The core admin SPLIT action assumes that the core being split is zookeeper aware. It will throw a NPE if invoked against a non-cloud solr setup. It should be fixed to work with non-cloud setups and documents in such an index should be distributed alternately into sub-indexes instead of using hashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13626753#comment-13626753 ] Shalin Shekhar Mangar commented on SOLR-3755: - Committed three changes: # Set update log to buffering mode before it is published (fixes bug with extra doc count on sub-shard) # Use deleteIndex=true while unloading sub-shard cores (if a sub-shard in construction state already exists at the start of the splitshard operation) # Made ChaosMonkeyShardSplitTest consistent with ShardSplitTest -- Use correct router and replica count, assert sub-shards are active, parent shards are inactive etc Anshum suggested over chat that we should think about combining ShardSplitTest and ChaosMonkeyShardSplit tests into one to avoid code duplication. I'll try to see if we can do that. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 4.3, 5.0 > > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4695) Fix core admin SPLIT action to be useful with non-cloud setups
[ https://issues.apache.org/jira/browse/SOLR-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-4695: Attachment: SOLR-4695.patch # Use CoreContainer.isZookeeperAware before trying to access cloud information # Split alternately if we don't know the real range of the core OR if we are not cloud aware # Test alternate distribution of docs in SolrIndexSplitterTest > Fix core admin SPLIT action to be useful with non-cloud setups > -- > > Key: SOLR-4695 > URL: https://issues.apache.org/jira/browse/SOLR-4695 > Project: Solr > Issue Type: Bug >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 4.3 > > Attachments: SOLR-4695.patch > > > The core admin SPLIT action assumes that the core being split is zookeeper > aware. It will throw a NPE if invoked against a non-cloud solr setup. > It should be fixed to work with non-cloud setups and documents in such an > index should be distributed alternately into sub-indexes instead of using > hashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4695) Fix core admin SPLIT action to be useful with non-cloud setups
[ https://issues.apache.org/jira/browse/SOLR-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-4695. - Resolution: Fixed > Fix core admin SPLIT action to be useful with non-cloud setups > -- > > Key: SOLR-4695 > URL: https://issues.apache.org/jira/browse/SOLR-4695 > Project: Solr > Issue Type: Bug >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 4.3 > > Attachments: SOLR-4695.patch > > > The core admin SPLIT action assumes that the core being split is zookeeper > aware. It will throw a NPE if invoked against a non-cloud solr setup. > It should be fixed to work with non-cloud setups and documents in such an > index should be distributed alternately into sub-indexes instead of using > hashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4685) JSON response write modification to support RAW JSON
[ https://issues.apache.org/jira/browse/SOLR-4685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628906#comment-13628906 ] Shalin Shekhar Mangar commented on SOLR-4685: - I'm sorry people. I mis-typed SOLR-4695 and put this issue's number in the commit and did it again while merging the fixed changelog entry into branch_4x. Please excuse these commits. > JSON response write modification to support RAW JSON > > > Key: SOLR-4685 > URL: https://issues.apache.org/jira/browse/SOLR-4685 > Project: Solr > Issue Type: Improvement >Reporter: Bill Bell > Attachments: SOLR-4685.patch > > > If the field ends with "_json" allow the field to return raw JSON. > For example the field, > office_json -- string > I already put into the field raw JSON already escaped. I want it to come with > no double quotes and not escaped. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4695) Fix core admin SPLIT action to be useful with non-cloud setups
[ https://issues.apache.org/jira/browse/SOLR-4695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13628915#comment-13628915 ] Shalin Shekhar Mangar commented on SOLR-4695: - Committed r1466860 to branch_4x (with the wrong jira issue# unfortunately) > Fix core admin SPLIT action to be useful with non-cloud setups > -- > > Key: SOLR-4695 > URL: https://issues.apache.org/jira/browse/SOLR-4695 > Project: Solr > Issue Type: Bug >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar > Fix For: 4.3 > > Attachments: SOLR-4695.patch > > > The core admin SPLIT action assumes that the core being split is zookeeper > aware. It will throw a NPE if invoked against a non-cloud solr setup. > It should be fixed to work with non-cloud setups and documents in such an > index should be distributed alternately into sub-indexes instead of using > hashes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-1853) SubPhraseQuery for matching and scoring sub phrase matches.
[
https://issues.apache.org/jira/browse/LUCENE-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631277#comment-13631277
]
Shalin Shekhar Mangar commented on LUCENE-1853:
---
Erick, SubPhraseQuery was written by Preetam for AOL Real Estate search. AFAIK,
no one is working actively on it.
> SubPhraseQuery for matching and scoring sub phrase matches.
> ---
>
> Key: LUCENE-1853
> URL: https://issues.apache.org/jira/browse/LUCENE-1853
> Project: Lucene - Core
> Issue Type: Improvement
> Components: core/search
> Environment: Lucene/Java
>Reporter: Preetam Rao
>Priority: Minor
> Attachments: LUCENE-1853.patch, LUCENE-1853.patch
>
>
> The goal is to give more control via configuration when searching using user
> entered queries against multiple fields where sub phrases have special
> significance.
> For a query like "homes in new york with swimming pool", if a document's
> field matches only "new york" it should get scored and it should get scored
> higher than two separate matches "new" and "york". Also, a 3 word sub phrase
> match must gets scored considerably higher than a 2 word sub phrase match.
> (boost factor should be configurable)
> Using shingles for this use case, means each field of each document needs to
> be indexed as shingles of all (1..N)-grams as well as the query. (Please
> correct me if I am wrong.)
> The query could also support
> - ignoring of idf and/or field norms, (so that factors outside the document
> don't influence scoring)
> - consider only the longest match (for example match on "new york" is scored
> and considered rather than "new" furniture and "york" city)
> - ignore duplicates ("new york" appearing twice or thrice does not make any
> difference)
> This kind of query could be combined with DisMax query. For example,
> something like solr's dismax request handler can be made to use this query
> where we run a user query as it is against all fields and configure each
> field with above configurations.
> I have also attached a patch with comments and test cases in case, my
> description is not clear enough. Would appreciate alternatives or feedback.
> Example Usage:
>
>// sub phrase config
> SubPhraseQuery.SubPhraseConfig conf = new
> SubPhraseQuery.SubPhraseConfig();
> conf.ignoreIdf = true;
> conf.ignoreFieldNorms = true;
> conf.matchOnlyLongest = true;
> conf.ignoreDuplicates = true;
> conf.phraseBoost = 2;
> // phrase query as usual
>SubPhraseQuery pq = new SubPhraseQuery();
>pq.add(new Term("f", term));
>pq.add(new Term("f", term));
> pq.setSubPhraseConf(conf);
> Hits hits = searcher.search(pq);
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1416) reduce contention in CoreContainer#getCore()
[ https://issues.apache.org/jira/browse/SOLR-1416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631334#comment-13631334 ] Shalin Shekhar Mangar commented on SOLR-1416: - Noble and I worked on LotsOfCores for AOL which had more than 20K cores per Solr instance. The top three factors slowing down solr for such use-cases were: # Opening IndexSearcher (our use-case had a 10:1 write/read ratio) - solved by opening searcher lazily # Loading/parsing IndexSchema and SolrConfig objects - solved by caching these objects # Contention in CoreContainer#getCore - solved by the attached patch At this later date, I don't have the data to support this change but it is worth benchmarking if someone is up for it. > reduce contention in CoreContainer#getCore() > > > Key: SOLR-1416 > URL: https://issues.apache.org/jira/browse/SOLR-1416 > Project: Solr > Issue Type: Improvement > Components: multicore >Reporter: Noble Paul >Priority: Minor > Attachments: SOLR-1416.patch > > > every call to CoreContainer#getCore() is synchronized . We should reduce the > contention . The writes are very infrequent and reads are frequent . How > about using a ReadWriterLock? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[
https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631384#comment-13631384
]
Shalin Shekhar Mangar commented on SOLR-3755:
-
bq. Anshum suggested over chat that we should think about combining
ShardSplitTest and ChaosMonkeyShardSplit tests into one to avoid code
duplication. I'll try to see if we can do that.
I've changed ChaosMonkeyShardSplitTest to extend ShardSplitTest so that we can
share most of the code. The ChaosMonkey test is not completely correct and I
intend to improve it.
bq. The original change around this made preRegister start taking a core rather
than a core descriptor. I'd like to work that out so it doesn't need to be the
case.
I'll revert the change to the preRegister method signature and find another way.
I've found two kinds of test failures of (ChaosMonkey)ShardSplitTest.
The first is because of the following sequence of events:
# A doc addition fails (because of the kill leader jetty command), client
throws an exception and therefore the docCount variable is not incremented
inside the index thread.
# However, the doc addition is recorded in the update logs (of the proxy node?)
and replayed on the new leader so in reality, the doc does get added to the
shard
# Split happens and we assert on docCounts being equal in the server which
fails because the server has the document that we have not counted.
This happens mostly with Lucene-Solr-Tests-4.x-Java6 builds. The bug is in the
tests and not in the split code. Following is the stack trace:
{code}
[junit4:junit4] 1> ERROR - 2013-04-14 14:24:27.697;
org.apache.solr.cloud.ChaosMonkeyShardSplitTest$1; Exception while adding doc
[junit4:junit4] 1> org.apache.solr.client.solrj.SolrServerException: No live
SolrServers available to handle this
request:[http://127.0.0.1:34203/h/y/collection1,
http://127.0.0.1:34304/h/y/collection1, http://127.0.0.1:34311/h/y/collection1,
http://127.0.0.1:34270/h/y/collection1]
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:333)
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.CloudSolrServer.request(CloudSolrServer.java:306)
[junit4:junit4] 1>at
org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117)
[junit4:junit4] 1>at
org.apache.solr.cloud.AbstractFullDistribZkTestBase.indexDoc(AbstractFullDistribZkTestBase.java:561)
[junit4:junit4] 1>at
org.apache.solr.cloud.ChaosMonkeyShardSplitTest.indexr(ChaosMonkeyShardSplitTest.java:434)
[junit4:junit4] 1>at
org.apache.solr.cloud.ChaosMonkeyShardSplitTest$1.run(ChaosMonkeyShardSplitTest.java:158)
[junit4:junit4] 1> Caused by: org.apache.solr.common.SolrException: Server at
http://127.0.0.1:34311/h/y/collection1 returned non ok status:503,
message:Service Unavailable
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:373)
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.LBHttpSolrServer.request(LBHttpSolrServer.java:264)
[junit4:junit4] 1>... 5 more
{code}
Perhaps we should check the exception message and continue to count such a
document?
The second kind of test failures are where a document add fails due to version
conflict. This exception is always seen just after the "updateshardstate" is
called to switch the shard states. Following is the relevant log:
{code}
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state invoked
for collection: collection1
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1
to inactive
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_0
to active
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_1
to active
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.873;
org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=
path=/update params={wt=javabin&version=2} {add=[169 (1432319507166134272)]} 0 2
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:juni
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631511#comment-13631511 ] Shalin Shekhar Mangar commented on SOLR-3755: - bq. Is this true for all the exceptions or just the one that follows this line? I wasn't able to reproduce this on my system running Java7. The error with the failing add doc happens with Java6 -- haven't seen it with any other version. I've seen the version conflict exception on java7 and java8. bq. Also, are these consistent failures? Yes but only on jenkins! I've had ec2 boxes running these tests all night and I haven't seen a failure in over 500 runs. These failures are very environment and timing dependent. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 4.3, 5.0 > > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-4693) Create a collections API to delete/cleanup a Slice
[ https://issues.apache.org/jira/browse/SOLR-4693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar reassigned SOLR-4693: --- Assignee: Shalin Shekhar Mangar > Create a collections API to delete/cleanup a Slice > -- > > Key: SOLR-4693 > URL: https://issues.apache.org/jira/browse/SOLR-4693 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Anshum Gupta >Assignee: Shalin Shekhar Mangar > Attachments: SOLR-4693.patch > > > Have a collections API that cleans up a given shard. > Among other places, this would be useful post the shard split call to manage > the parent/original slice. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4566) clusterstate#getSlices and getSlicesMap should always return all slices.
[ https://issues.apache.org/jira/browse/SOLR-4566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13631733#comment-13631733 ] Shalin Shekhar Mangar commented on SOLR-4566: - The attached patch was committed to trunk and branch_4x along with the shard split feature. If people are happy with the API proposed in the patch then we can mark this issue as resolved. > clusterstate#getSlices and getSlicesMap should always return all slices. > > > Key: SOLR-4566 > URL: https://issues.apache.org/jira/browse/SOLR-4566 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Mark Miller >Assignee: Mark Miller > Fix For: 4.3, 5.0 > > Attachments: SOLR-4566.patch > > > I'm not sure I really like this getSlices vs getAllSclies - it's kind of easy > to get in trouble right now I think. Some spots that we clearly want to call > getAllSlices got left with getSlices. It's almost surprising that getSlices > only returns active replicas - it should probably at least be called > getSlicesWithActive or something more explicit. But for the first step, we > should just fix the various mis calls. > There are a couple problems around the mis calls, the most severe probably > being that you can lose shards that are not active from the clusterstate.json > given the right races. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13637505#comment-13637505 ] Shalin Shekhar Mangar commented on SOLR-3755: - I haven't seen the test failure due to extra document after increasing read timeout values in the test. Now that 4.3 is about to release with this feature, I'm going to mark this issue as resolved. > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 4.3, 5.0 > > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-3755) shard splitting
[ https://issues.apache.org/jira/browse/SOLR-3755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-3755. - Resolution: Fixed This feature will be released with Solr 4.3 > shard splitting > --- > > Key: SOLR-3755 > URL: https://issues.apache.org/jira/browse/SOLR-3755 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Assignee: Shalin Shekhar Mangar > Fix For: 4.3, 5.0 > > Attachments: SOLR-3755-combined.patch, > SOLR-3755-combinedWithReplication.patch, SOLR-3755-CoreAdmin.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, SOLR-3755.patch, > SOLR-3755.patch, SOLR-3755.patch, SOLR-3755-testSplitter.patch, > SOLR-3755-testSplitter.patch > > > We can currently easily add replicas to handle increases in query volume, but > we should also add a way to add additional shards dynamically by splitting > existing shards. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-4744) Version conflict error during shard split test
Shalin Shekhar Mangar created SOLR-4744:
---
Summary: Version conflict error during shard split test
Key: SOLR-4744
URL: https://issues.apache.org/jira/browse/SOLR-4744
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 4.3
Reporter: Shalin Shekhar Mangar
Assignee: Shalin Shekhar Mangar
Priority: Minor
Fix For: 4.4
ShardSplitTest fails sometimes with the following error:
{code}
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state invoked
for collection: collection1
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1
to inactive
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_0
to active
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.861;
org.apache.solr.cloud.Overseer$ClusterStateUpdater; Update shard state shard1_1
to active
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.873;
org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=
path=/update params={wt=javabin&version=2} {add=[169 (1432319507166134272)]} 0 2
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.877;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged path:/clusterstate.json,
has occurred - updating... (live nodes size: 5)
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.884;
org.apache.solr.update.processor.LogUpdateProcessor;
[collection1_shard1_1_replica1] webapp= path=/update
params={distrib.from=http://127.0.0.1:41028/collection1/&update.distrib=FROMLEADER&wt=javabin&distrib.from.parent=shard1&version=2}
{} 0 1
[junit4:junit4] 1> INFO - 2013-04-14 19:05:26.885;
org.apache.solr.update.processor.LogUpdateProcessor; [collection1] webapp=
path=/update
params={distrib.from=http://127.0.0.1:41028/collection1/&update.distrib=FROMLEADER&wt=javabin&distrib.from.parent=shard1&version=2}
{add=[169 (1432319507173474304)]} 0 2
[junit4:junit4] 1> ERROR - 2013-04-14 19:05:26.885;
org.apache.solr.common.SolrException; shard update error StdNode:
http://127.0.0.1:41028/collection1_shard1_1_replica1/:org.apache.solr.common.SolrException:
version conflict for 169 expected=1432319507173474304 actual=-1
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:404)
[junit4:junit4] 1>at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181)
[junit4:junit4] 1>at
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:332)
[junit4:junit4] 1>at
org.apache.solr.update.SolrCmdDistributor$1.call(SolrCmdDistributor.java:306)
[junit4:junit4] 1>at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 1>at
java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 1>at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
[junit4:junit4] 1>at
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
[junit4:junit4] 1>at
java.util.concurrent.FutureTask.run(FutureTask.java:166)
[junit4:junit4] 1>at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
[junit4:junit4] 1>at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit4:junit4] 1>at java.lang.Thread.run(Thread.java:679)
[junit4:junit4
[jira] [Created] (SOLR-4745) Do not pass a SolrCore in ZkController.preRegister()
Shalin Shekhar Mangar created SOLR-4745: --- Summary: Do not pass a SolrCore in ZkController.preRegister() Key: SOLR-4745 URL: https://issues.apache.org/jira/browse/SOLR-4745 Project: Solr Issue Type: Task Components: SolrCloud Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Priority: Minor Fix For: 4.4 The shard splitting feature changed the method definition of ZkController#preRegister to accept a SolrCore instead of CoreDescriptor. >From the comments in SOLR-3755: bq. Set update log to buffering mode before it is published (fixes bug with extra doc count on sub-shard) bq. Regarding those changes - I'd really like to find another way to do that. The original change around this made preRegister start taking a core rather than a core descriptor. I'd like to work that out so it doesn't need to be the case. That is where the core will find out some of it's properties (shard id, core node name, perhaps more in the future). It would be nice if the core init code had access to this information - so it would be nice if we could call preRegister (or some refactored version) before actually creating the SolrCore. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4747) DIH postgres connections are not closed
[ https://issues.apache.org/jira/browse/SOLR-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13638031#comment-13638031 ] Shalin Shekhar Mangar commented on SOLR-4747: - It sounds like you want DIH to close all connections per root entity as soon as all rows have been processed. This way, when DIH goes to the next root entity, connections for the previous entity's data sources would have been closed. If so, the title of the issue should be changed because DIH does close connections but not when you expect. > DIH postgres connections are not closed > --- > > Key: SOLR-4747 > URL: https://issues.apache.org/jira/browse/SOLR-4747 > Project: Solr > Issue Type: Bug > Components: contrib - DataImportHandler >Affects Versions: 4.2 > Environment: Solr on CentOs 6.4 within jboss, debian 7.0 postgres > server >Reporter: Remko Kuipers > > I use DIH to create an index for a number of postgres databases. These > databases all contain an identical schema/table setup. The config looks like > repeated for each > datasource. The readonly flag is set to true on the subentity objects. For > each database solr creates three connections. These connections remain active > as "idle in transaction" until the indexer is finished. As the number of > databases times three exceeds my max_connections setting in postgres the > indexer inevitably fails halfway through with a no more connections available > error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-2356) indexing using DataImportHandler does not use entire CPU capacities
[ https://issues.apache.org/jira/browse/SOLR-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13642576#comment-13642576 ] Shalin Shekhar Mangar commented on SOLR-2356: - bq. In my opinion, DIH should be completely redesigned as a standalone webapp. It is a major design flaw that it is a RequestHandler within a Solr Core/collection. Actually, DIH started as a standalone webapp inside AOL. We changed it because we didn't want to duplicate the schema in two places and also because we wanted to have it available by default in Solr installations. Another web app means you need to procure hardware, plan capacity/failover, create firewall holes etc bq. As a standalone web app it could easily be deplyed on its own, talk to multiple collections and be parallellized. Talking to multiple collections was never a goal for DIH -- I'm not sure what value it will bring. The multi-threading support in DIH can use a lot of improvement for sure. > indexing using DataImportHandler does not use entire CPU capacities > --- > > Key: SOLR-2356 > URL: https://issues.apache.org/jira/browse/SOLR-2356 > Project: Solr > Issue Type: Improvement > Components: update >Affects Versions: 4.0-ALPHA > Environment: intel xeon processor (4 cores), Debian Linux Lenny, > OpenJDK 64bits server v1.6.0 >Reporter: colby >Priority: Minor > Labels: test > Original Estimate: 168h > Remaining Estimate: 168h > > When I use a DataImportHandler to index a large number of documents (~35M), > cpu usage doesn't go over than 100% cpu (i.e. just one core). > When I configure 4 threads for the tag, the cpu usage is splitted to > 25% per core but never use 400% of cpu (i.e 100% of the 4 cores) > I use solr embedded with jetty server. > Is there a way to tune this feature in order to use all cores and improve > indexing performances ? > Because for the moment, an extra script (PHP) gives better indexing > performances than DIH. > thanks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4776) Solrj doesn't return "between" count in range facets
[ https://issues.apache.org/jira/browse/SOLR-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-4776: Attachment: SOLR-4776.patch > Solrj doesn't return "between" count in range facets > > > Key: SOLR-4776 > URL: https://issues.apache.org/jira/browse/SOLR-4776 > Project: Solr > Issue Type: Bug > Components: clients - java >Affects Versions: 4.2 >Reporter: Philip K. Warren >Priority: Minor > Attachments: SOLR-4776.patch, SOLR-4776.patch > > > The current RangeFacet class exposes the "before" and "after" counts, however > it doesn't expose the "between" count, which can be returned if > facet.range.other includes "between" or is set to "all". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-4776) Solrj doesn't return "between" count in range facets
[ https://issues.apache.org/jira/browse/SOLR-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-4776: Attachment: (was: SOLR-4776.patch) > Solrj doesn't return "between" count in range facets > > > Key: SOLR-4776 > URL: https://issues.apache.org/jira/browse/SOLR-4776 > Project: Solr > Issue Type: Bug > Components: clients - java >Affects Versions: 4.2 >Reporter: Philip K. Warren >Priority: Minor > Attachments: SOLR-4776.patch > > > The current RangeFacet class exposes the "before" and "after" counts, however > it doesn't expose the "between" count, which can be returned if > facet.range.other includes "between" or is set to "all". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4776) Solrj doesn't return "between" count in range facets
[ https://issues.apache.org/jira/browse/SOLR-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13645028#comment-13645028 ] Shalin Shekhar Mangar commented on SOLR-4776: - Looks good. I'll commit shortly. > Solrj doesn't return "between" count in range facets > > > Key: SOLR-4776 > URL: https://issues.apache.org/jira/browse/SOLR-4776 > Project: Solr > Issue Type: Bug > Components: clients - java >Affects Versions: 4.2 >Reporter: Philip K. Warren >Priority: Minor > Attachments: SOLR-4776.patch > > > The current RangeFacet class exposes the "before" and "after" counts, however > it doesn't expose the "between" count, which can be returned if > facet.range.other includes "between" or is set to "all". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-4776) Solrj doesn't return "between" count in range facets
[ https://issues.apache.org/jira/browse/SOLR-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-4776. - Resolution: Fixed Fix Version/s: 4.4 5.0 Assignee: Shalin Shekhar Mangar > Solrj doesn't return "between" count in range facets > > > Key: SOLR-4776 > URL: https://issues.apache.org/jira/browse/SOLR-4776 > Project: Solr > Issue Type: Bug > Components: clients - java >Affects Versions: 4.2 >Reporter: Philip K. Warren >Assignee: Shalin Shekhar Mangar >Priority: Minor > Fix For: 5.0, 4.4 > > Attachments: SOLR-4776.patch > > > The current RangeFacet class exposes the "before" and "after" counts, however > it doesn't expose the "between" count, which can be returned if > facet.range.other includes "between" or is set to "all". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13369) TriLevelCompositeIdRoutingTest failure: same route prefix maped to multiple shards
[
https://issues.apache.org/jira/browse/SOLR-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16812180#comment-16812180
]
Shalin Shekhar Mangar commented on SOLR-13369:
--
My understanding in the case of {{app9/2!user32!doc58}} is that:
* 2 bits are taken from the 32 bit hash of app9
* 8 bits are taken from the 32 bit hash of user32
* 22 bits are taken from the 32 bit hash of doc58
Going back to what you wrote in SOLR-13210, I think you're right. When the bits
is specified neither app nor app!user is guaranteed to be limited to a single
shard. I don't really understand this router very well. Perhaps [~anshumg] can
help?
> TriLevelCompositeIdRoutingTest failure: same route prefix maped to multiple
> shards
> --
>
> Key: SOLR-13369
> URL: https://issues.apache.org/jira/browse/SOLR-13369
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Hoss Man
>Priority: Major
> Attachments:
> TriLevelCompositeIdRoutingTest-failure-with-debug-log.txt,
> thetaphi_Lucene-Solr-8.x-Linux_342.log.txt
>
>
> thetaphi's 8x jenkins job just identified a reproducing seed that causes
> TriLevelCompositeIdRoutingTest to fail after detecting 2 docs with matching
> route prefixes on different shards...
> {noformat}
>[junit4] 2> NOTE: reproduce with: ant test
> -Dtestcase=TriLevelCompositeIdRoutingTest -Dtests.method=test
> -Dtests.seed=A6B6F0104FE6018F -Dtests.multiplier=3 -Dtests.slow=true
> -Dtests.locale=sr-Latn -Dtests.timezone=Pacific/Tongatapu
> -Dtests.asserts=true -Dtests.file.encoding=US-ASCII
>[junit4] FAILURE 9.38s J0 | TriLevelCompositeIdRoutingTest.test <<<
>[junit4]> Throwable #1: org.junit.ComparisonFailure: routePrefix
> app9/2!user32 found in multiple shards expected: but was:
>[junit4]>at
> __randomizedtesting.SeedInfo.seed([A6B6F0104FE6018F:2EE2CFCAE11A6C77]:0)
>[junit4]>at
> org.apache.solr.cloud.TriLevelCompositeIdRoutingTest.test(TriLevelCompositeIdRoutingTest.java:122)
> {noformat}
> It's possible this is just a bug I introduced in SOLR-13210 due to a
> missunderstanding in how routePrefixes that use a bit mask (ie: {{/2}} in the
> assertion failure) are expected to work -- but I thought i had that squared
> away based on shalin's feedback in SOLR-13210
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13401) Metrics showing wring 'spins' property after override
[
https://issues.apache.org/jira/browse/SOLR-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818573#comment-16818573
]
Shalin Shekhar Mangar commented on SOLR-13401:
--
Ah, I thought that the metrics API will show the computed value if the
parameter is present. I guess there is no way to verify the settings took
effect until we actually create a core and check the default values of
ConcurrentMergeScheduler? How about we show the overriden value for spins
instead? After all, that is what is actually being used by Lucene.
> Metrics showing wring 'spins' property after override
> -
>
> Key: SOLR-13401
> URL: https://issues.apache.org/jira/browse/SOLR-13401
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
>Affects Versions: 8.0
>Reporter: Amrit Sarkar
>Priority: Major
> Labels: metrics
>
> Even after setting required param to "false" acc to documentation –
> https://lucene.apache.org/solr/guide/7_4/taking-solr-to-production.html#dynamic-defaults-for-concurrentmergescheduler
> Alternatively, the boolean system property {{lucene.cms.override_spins}} can
> be set in the SOLR_OPTS variable in the include file to override the
> auto-detected value. Similarly, the system property
> {{lucene.cms.override_core_count}} can be set to the number of CPU cores to
> override the auto-detected processor count.
> Container.fs.spins and Core.fs.spins are calculated as "true".
> {code}
> {
> "responseHeader":{
> "status":0,
> "QTime":90},
> "metrics":{
> "solr.jetty":{
> "org.eclipse.jetty.server.handler.DefaultHandler.1xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses":{
> "count":749,
> "meanRate":0.26577854703424414,
> "1minRate":0.20563648170407736,
> "5minRate":0.2340577654572216,
> "15minRate":0.24729247425529033},
> "org.eclipse.jetty.server.handler.DefaultHandler.3xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.4xx-responses":{
> "count":1,
> "meanRate":3.548445163413177E-4,
> "1minRate":6.64685036699106E-18,
> "5minRate":2.7733971887474625E-6,
> "15minRate":1.0450429716535352E-4},
> "org.eclipse.jetty.server.handler.DefaultHandler.5xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.active-dispatches":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-requests":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-suspended":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.async-dispatches":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.async-timeouts":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.connect-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.delete-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.dispatches":{
> "count":750,
> "meanRate":0.26613247165752213,
> "1minRate":0.20563648170407736,
> "5minRate":0.23406053885441036,
> "15minRate":0.24739697855245568,
> "min_ms":1.0,
> "max_ms":8231.0,
> "mean_ms":1.9138611068896936,
> "median_ms":1.0,
> "stddev_ms":4.227101567072412,
> "p75_ms":2.0,
> "p95_ms":2.0,
> "p99_ms":34.0,
> "p999_ms":48.0}
[jira] [Commented] (SOLR-13401) Metrics showing wring 'spins' property after override
[
https://issues.apache.org/jira/browse/SOLR-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818755#comment-16818755
]
Shalin Shekhar Mangar commented on SOLR-13401:
--
Fair enough, I will mark this issue as invalid.
> Metrics showing wring 'spins' property after override
> -
>
> Key: SOLR-13401
> URL: https://issues.apache.org/jira/browse/SOLR-13401
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
>Affects Versions: 8.0
>Reporter: Amrit Sarkar
>Priority: Major
> Labels: metrics
>
> Even after setting required param to "false" acc to documentation –
> https://lucene.apache.org/solr/guide/7_4/taking-solr-to-production.html#dynamic-defaults-for-concurrentmergescheduler
> Alternatively, the boolean system property {{lucene.cms.override_spins}} can
> be set in the SOLR_OPTS variable in the include file to override the
> auto-detected value. Similarly, the system property
> {{lucene.cms.override_core_count}} can be set to the number of CPU cores to
> override the auto-detected processor count.
> Container.fs.spins and Core.fs.spins are calculated as "true".
> {code}
> {
> "responseHeader":{
> "status":0,
> "QTime":90},
> "metrics":{
> "solr.jetty":{
> "org.eclipse.jetty.server.handler.DefaultHandler.1xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses":{
> "count":749,
> "meanRate":0.26577854703424414,
> "1minRate":0.20563648170407736,
> "5minRate":0.2340577654572216,
> "15minRate":0.24729247425529033},
> "org.eclipse.jetty.server.handler.DefaultHandler.3xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.4xx-responses":{
> "count":1,
> "meanRate":3.548445163413177E-4,
> "1minRate":6.64685036699106E-18,
> "5minRate":2.7733971887474625E-6,
> "15minRate":1.0450429716535352E-4},
> "org.eclipse.jetty.server.handler.DefaultHandler.5xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.active-dispatches":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-requests":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-suspended":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.async-dispatches":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.async-timeouts":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.connect-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.delete-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.dispatches":{
> "count":750,
> "meanRate":0.26613247165752213,
> "1minRate":0.20563648170407736,
> "5minRate":0.23406053885441036,
> "15minRate":0.24739697855245568,
> "min_ms":1.0,
> "max_ms":8231.0,
> "mean_ms":1.9138611068896936,
> "median_ms":1.0,
> "stddev_ms":4.227101567072412,
> "p75_ms":2.0,
> "p95_ms":2.0,
> "p99_ms":34.0,
> "p999_ms":48.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.get-requests":{
> "count":427,
> "meanRate":0.15151854520088134,
> "1minRate":0.12770720959194207,
> "5minRate":0.13538027663316224,
> "15minRate":0.13928403821024407,
> "min_ms":1.0,
> "max_ms":175
[jira] [Resolved] (SOLR-13401) Metrics showing wring 'spins' property after override
[
https://issues.apache.org/jira/browse/SOLR-13401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar resolved SOLR-13401.
--
Resolution: Invalid
> Metrics showing wring 'spins' property after override
> -
>
> Key: SOLR-13401
> URL: https://issues.apache.org/jira/browse/SOLR-13401
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
>Affects Versions: 8.0
>Reporter: Amrit Sarkar
>Priority: Major
> Labels: metrics
>
> Even after setting required param to "false" acc to documentation –
> https://lucene.apache.org/solr/guide/7_4/taking-solr-to-production.html#dynamic-defaults-for-concurrentmergescheduler
> Alternatively, the boolean system property {{lucene.cms.override_spins}} can
> be set in the SOLR_OPTS variable in the include file to override the
> auto-detected value. Similarly, the system property
> {{lucene.cms.override_core_count}} can be set to the number of CPU cores to
> override the auto-detected processor count.
> Container.fs.spins and Core.fs.spins are calculated as "true".
> {code}
> {
> "responseHeader":{
> "status":0,
> "QTime":90},
> "metrics":{
> "solr.jetty":{
> "org.eclipse.jetty.server.handler.DefaultHandler.1xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.2xx-responses":{
> "count":749,
> "meanRate":0.26577854703424414,
> "1minRate":0.20563648170407736,
> "5minRate":0.2340577654572216,
> "15minRate":0.24729247425529033},
> "org.eclipse.jetty.server.handler.DefaultHandler.3xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.4xx-responses":{
> "count":1,
> "meanRate":3.548445163413177E-4,
> "1minRate":6.64685036699106E-18,
> "5minRate":2.7733971887474625E-6,
> "15minRate":1.0450429716535352E-4},
> "org.eclipse.jetty.server.handler.DefaultHandler.5xx-responses":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.active-dispatches":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-requests":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.active-suspended":0,
> "org.eclipse.jetty.server.handler.DefaultHandler.async-dispatches":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.async-timeouts":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.connect-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.delete-requests":{
> "count":0,
> "meanRate":0.0,
> "1minRate":0.0,
> "5minRate":0.0,
> "15minRate":0.0,
> "min_ms":0.0,
> "max_ms":0.0,
> "mean_ms":0.0,
> "median_ms":0.0,
> "stddev_ms":0.0,
> "p75_ms":0.0,
> "p95_ms":0.0,
> "p99_ms":0.0,
> "p999_ms":0.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.dispatches":{
> "count":750,
> "meanRate":0.26613247165752213,
> "1minRate":0.20563648170407736,
> "5minRate":0.23406053885441036,
> "15minRate":0.24739697855245568,
> "min_ms":1.0,
> "max_ms":8231.0,
> "mean_ms":1.9138611068896936,
> "median_ms":1.0,
> "stddev_ms":4.227101567072412,
> "p75_ms":2.0,
> "p95_ms":2.0,
> "p99_ms":34.0,
> "p999_ms":48.0},
> "org.eclipse.jetty.server.handler.DefaultHandler.get-requests":{
> "count":427,
> "meanRate":0.15151854520088134,
> "1minRate":0.12770720959194207,
> "5minRate":0.13538027663316224,
> "15minRate":0.13928403821024407,
> "min_ms":1.0,
> "max_ms":1759.0,
> "mean_ms":2.059088748457511,
> "median_ms":1.0,
>
[jira] [Assigned] (SOLR-13392) Unable to start prometheus-exporter in 7x branch
[ https://issues.apache.org/jira/browse/SOLR-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar reassigned SOLR-13392: Assignee: Shalin Shekhar Mangar > Unable to start prometheus-exporter in 7x branch > > > Key: SOLR-13392 > URL: https://issues.apache.org/jira/browse/SOLR-13392 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Affects Versions: 7.7.2 >Reporter: Karl Stoney >Assignee: Shalin Shekhar Mangar >Priority: Major > > Hi, > prometheus-exporter doesn't start in branch 7x on commit > 7dfe1c093b65f77407c2df4c2a1120a213aef166, it does work on > 26b498d0a9d25626a15e25b0cf97c8339114263a so something has changed between > those two commits causing this. > I am presuming it is > https://github.com/apache/lucene-solr/commit/e1eeafb5dc077976646b06f4cba4d77534963fa9#diff-3f7b27f0f087632739effa2aa508d77eR34 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/lucene/util/IOUtils > at > org.apache.solr.core.SolrResourceLoader.close(SolrResourceLoader.java:881) > at > org.apache.solr.prometheus.exporter.SolrExporter.loadMetricsConfiguration(SolrExporter.java:221) > at > org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:205) > Caused by: java.lang.ClassNotFoundException: org.apache.lucene.util.IOUtils > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 3 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13392) Unable to start prometheus-exporter in 7x branch
[ https://issues.apache.org/jira/browse/SOLR-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819716#comment-16819716 ] Shalin Shekhar Mangar commented on SOLR-13392: -- Ok I can reproduce this easily. For some reason, our ivy and build.xml configurations add lucene-core as a dependency (perhaps transitively) to the prometheus-exporter contrib. But the runtime does not include that jar. I'll try to see if I can get the builds to fail because of this dependency and fix that first and then remove the lucene dependency from prometheus as it does not need it. > Unable to start prometheus-exporter in 7x branch > > > Key: SOLR-13392 > URL: https://issues.apache.org/jira/browse/SOLR-13392 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Affects Versions: 7.7.2 >Reporter: Karl Stoney >Assignee: Shalin Shekhar Mangar >Priority: Major > > Hi, > prometheus-exporter doesn't start in branch 7x on commit > 7dfe1c093b65f77407c2df4c2a1120a213aef166, it does work on > 26b498d0a9d25626a15e25b0cf97c8339114263a so something has changed between > those two commits causing this. > I am presuming it is > https://github.com/apache/lucene-solr/commit/e1eeafb5dc077976646b06f4cba4d77534963fa9#diff-3f7b27f0f087632739effa2aa508d77eR34 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/lucene/util/IOUtils > at > org.apache.solr.core.SolrResourceLoader.close(SolrResourceLoader.java:881) > at > org.apache.solr.prometheus.exporter.SolrExporter.loadMetricsConfiguration(SolrExporter.java:221) > at > org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:205) > Caused by: java.lang.ClassNotFoundException: org.apache.lucene.util.IOUtils > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 3 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13392) Unable to start prometheus-exporter in 7x branch
[
https://issues.apache.org/jira/browse/SOLR-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819747#comment-16819747
]
Shalin Shekhar Mangar commented on SOLR-13392:
--
Okay, so it is a little more tricky.
The exporter adds solr core as a dependency in its build and adds lucene's
analysis jar in the distribution so that it can access the lucene's
ResourceLoader (transitively from the SolrResourceLoader). This is why ant test
does not catch the problem. The resource loader calls IOUtils.close (which is
in lucene-core).
Before SOLR-13234, the loader's close method was never called(?) which is why
this dependency miss was not spotted. But the rabbit hole goes deeper, after
SOLR-13234, the exporter uses more of Solr core's dependencies such as guava
cache so it is not sufficient to just add lucene-core.jar. It is quite a mess,
I admit. It is easy to just add all of solr runtime dependencies on the class
path by adding {{server/solr-webapp/webapp/WEB-INF/lib}} to the classpath.
Other scripts such as zkCli do the same. Are there any objections on doing that?
> Unable to start prometheus-exporter in 7x branch
>
>
> Key: SOLR-13392
> URL: https://issues.apache.org/jira/browse/SOLR-13392
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: metrics
>Affects Versions: 7.7.2
>Reporter: Karl Stoney
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-13392.patch
>
>
> Hi,
> prometheus-exporter doesn't start in branch 7x on commit
> 7dfe1c093b65f77407c2df4c2a1120a213aef166, it does work on
> 26b498d0a9d25626a15e25b0cf97c8339114263a so something has changed between
> those two commits causing this.
> I am presuming it is
> https://github.com/apache/lucene-solr/commit/e1eeafb5dc077976646b06f4cba4d77534963fa9#diff-3f7b27f0f087632739effa2aa508d77eR34
> Exception in thread "main" java.lang.NoClassDefFoundError:
> org/apache/lucene/util/IOUtils
> at
> org.apache.solr.core.SolrResourceLoader.close(SolrResourceLoader.java:881)
> at
> org.apache.solr.prometheus.exporter.SolrExporter.loadMetricsConfiguration(SolrExporter.java:221)
> at
> org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:205)
> Caused by: java.lang.ClassNotFoundException: org.apache.lucene.util.IOUtils
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 3 more
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13392) Unable to start prometheus-exporter in 7x branch
[ https://issues.apache.org/jira/browse/SOLR-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-13392: - Attachment: SOLR-13392.patch > Unable to start prometheus-exporter in 7x branch > > > Key: SOLR-13392 > URL: https://issues.apache.org/jira/browse/SOLR-13392 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Affects Versions: 7.7.2 >Reporter: Karl Stoney >Assignee: Shalin Shekhar Mangar >Priority: Major > Attachments: SOLR-13392.patch > > > Hi, > prometheus-exporter doesn't start in branch 7x on commit > 7dfe1c093b65f77407c2df4c2a1120a213aef166, it does work on > 26b498d0a9d25626a15e25b0cf97c8339114263a so something has changed between > those two commits causing this. > I am presuming it is > https://github.com/apache/lucene-solr/commit/e1eeafb5dc077976646b06f4cba4d77534963fa9#diff-3f7b27f0f087632739effa2aa508d77eR34 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/lucene/util/IOUtils > at > org.apache.solr.core.SolrResourceLoader.close(SolrResourceLoader.java:881) > at > org.apache.solr.prometheus.exporter.SolrExporter.loadMetricsConfiguration(SolrExporter.java:221) > at > org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:205) > Caused by: java.lang.ClassNotFoundException: org.apache.lucene.util.IOUtils > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 3 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13392) Unable to start prometheus-exporter in 7x branch
[ https://issues.apache.org/jira/browse/SOLR-13392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819748#comment-16819748 ] Shalin Shekhar Mangar commented on SOLR-13392: -- I've attached a trivial patch that adds all jars in solr-webapp/webapp/WEB-INF/lib to the classpath in the exporter's bin scripts. > Unable to start prometheus-exporter in 7x branch > > > Key: SOLR-13392 > URL: https://issues.apache.org/jira/browse/SOLR-13392 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: metrics >Affects Versions: 7.7.2 >Reporter: Karl Stoney >Assignee: Shalin Shekhar Mangar >Priority: Major > Attachments: SOLR-13392.patch > > > Hi, > prometheus-exporter doesn't start in branch 7x on commit > 7dfe1c093b65f77407c2df4c2a1120a213aef166, it does work on > 26b498d0a9d25626a15e25b0cf97c8339114263a so something has changed between > those two commits causing this. > I am presuming it is > https://github.com/apache/lucene-solr/commit/e1eeafb5dc077976646b06f4cba4d77534963fa9#diff-3f7b27f0f087632739effa2aa508d77eR34 > Exception in thread "main" java.lang.NoClassDefFoundError: > org/apache/lucene/util/IOUtils > at > org.apache.solr.core.SolrResourceLoader.close(SolrResourceLoader.java:881) > at > org.apache.solr.prometheus.exporter.SolrExporter.loadMetricsConfiguration(SolrExporter.java:221) > at > org.apache.solr.prometheus.exporter.SolrExporter.main(SolrExporter.java:205) > Caused by: java.lang.ClassNotFoundException: org.apache.lucene.util.IOUtils > at java.net.URLClassLoader.findClass(URLClassLoader.java:382) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 3 more -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13737) Lead with SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16921978#comment-16921978 ] Shalin Shekhar Mangar commented on SOLR-13737: -- +1 > Lead with SolrCloud > --- > > Key: SOLR-13737 > URL: https://issues.apache.org/jira/browse/SOLR-13737 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Affects Versions: master (9.0) >Reporter: Marcus Eagan >Priority: Trivial > Fix For: master (9.0) > > > Based on some of the unnecessary and non-constructive criticism I have heard > that SolrCloud is an after thought in 2019, which is totally not true, I > decided it might be better if we moved it up ahead of standalone Solr in the > README. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13737) Lead with SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-13737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16922181#comment-16922181 ] Shalin Shekhar Mangar commented on SOLR-13737: -- PR is at https://github.com/apache/lucene-solr/pull/853 > Lead with SolrCloud > --- > > Key: SOLR-13737 > URL: https://issues.apache.org/jira/browse/SOLR-13737 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Affects Versions: master (9.0) >Reporter: Marcus Eagan >Priority: Trivial > Fix For: master (9.0) > > > Based on some of the unnecessary and non-constructive criticism I have heard > that SolrCloud is an after thought in 2019, which is totally not true, I > decided it might be better if we moved it up ahead of standalone Solr in the > README. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12940) Optimize and expunge deletes should execute only on the leader instead of all replicas of the collection
Shalin Shekhar Mangar created SOLR-12940: Summary: Optimize and expunge deletes should execute only on the leader instead of all replicas of the collection Key: SOLR-12940 URL: https://issues.apache.org/jira/browse/SOLR-12940 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Reporter: Shalin Shekhar Mangar Fix For: 7.6, master (8.0) Optimize and expunge deletes are broadcasted to all replicas of the collection (even to replicas of inactive shards!) but they don't need to be. We can optimize by only executing such commands on the leader and have the replicas pull the index over the network when ready. Synchronizing replication recovery to happen after optimize completes was trickier in the past when all we had was HTTP requests but now we have the terms mechanism which goes via ZK and can be relied on. This gives us a nice way to index/update fully and then call optimize/expunge deletes only on the leader while replicas continue to serve query traffic without disruption. This use-case will also need the ability to route queries only to replicas and not to the leader. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12023) Autoscaling policy engine shuffles replicas needlessly and can also suggest nonexistent replicas to be moved
[ https://issues.apache.org/jira/browse/SOLR-12023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16668505#comment-16668505 ] Shalin Shekhar Mangar commented on SOLR-12023: -- [~noble.paul] - I see that the commit only fixes suggesting non-existent cores problem. What about the needlessly shuffling replicas -- is that already fixed? Also, where is the corresponding commit to master? > Autoscaling policy engine shuffles replicas needlessly and can also suggest > nonexistent replicas to be moved > > > Key: SOLR-12023 > URL: https://issues.apache.org/jira/browse/SOLR-12023 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Noble Paul >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-11066-failing.patch, SOLR-12023.patch > > > A test that I wrote in SOLR-11066 found the following problem: > Cluster: 2 nodes > Collection: 1 shard, 3 replicas, maxShardsPerNode=5 > No autoscaling policy or preference applied > When the trigger runs, the computed plan needlessly shuffles all three > replicas and then proceeds to return suggestions with only numbers as core > names. These cores do not exist. I found that these numbers are generated > internally by the framework as placeholders for moved cores for further > calculations. They should never ever be suggested to the users. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12739) Make autoscaling policy based replica placement the default strategy for placing replicas
[ https://issues.apache.org/jira/browse/SOLR-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-12739. -- Resolution: Fixed > Make autoscaling policy based replica placement the default strategy for > placing replicas > - > > Key: SOLR-12739 > URL: https://issues.apache.org/jira/browse/SOLR-12739 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-12739.patch, SOLR-12739.patch, SOLR-12739.patch, > SOLR-12739.patch, SOLR-12739.patch > > > Today the default placement strategy is the same one used since Solr 4.x > which is to select nodes on a round robin fashion. I propose to make the > autoscaling policy based replica placement as the default policy for placing > replicas. > This is related to SOLR-12648 where even though we have default cluster > preferences, we don't use them unless a policy is also configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-12845:
-
Attachment: SOLR-12845.patch
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch, SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672964#comment-16672964
]
Shalin Shekhar Mangar commented on SOLR-12845:
--
Updated patch that applies to latest master and fixes the failure in
{{TestUtilizeNode}}
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch, SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13082) A trigger that creates trigger events more frequently than the cool down period can starve other triggers
Shalin Shekhar Mangar created SOLR-13082: Summary: A trigger that creates trigger events more frequently than the cool down period can starve other triggers Key: SOLR-13082 URL: https://issues.apache.org/jira/browse/SOLR-13082 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: AutoScaling Reporter: Shalin Shekhar Mangar Fix For: master (8.0), 7.7 A trigger that creates frequent events (such as ScheduledTrigger) with period less than the cool down period can starve all other triggers. This is because we execute triggers in the same order each time so if the scheduled trigger happens to be the first in the list then after every cool down period, the scheduled trigger can create another event and the cycle repeats. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13082) A trigger that creates trigger events more frequently than the cool down period can starve other triggers
[ https://issues.apache.org/jira/browse/SOLR-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-13082: - Attachment: SOLR-13082.patch > A trigger that creates trigger events more frequently than the cool down > period can starve other triggers > - > > Key: SOLR-13082 > URL: https://issues.apache.org/jira/browse/SOLR-13082 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.7 > > Attachments: SOLR-13082.patch > > > A trigger that creates frequent events (such as ScheduledTrigger) with period > less than the cool down period can starve all other triggers. This is because > we execute triggers in the same order each time so if the scheduled trigger > happens to be the first in the list then after every cool down period, the > scheduled trigger can create another event and the cycle repeats. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13082) A trigger that creates trigger events more frequently than the cool down period can starve other triggers
[ https://issues.apache.org/jira/browse/SOLR-13082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16725616#comment-16725616 ] Shalin Shekhar Mangar commented on SOLR-13082: -- Here's a patch which shuffles triggers on resume to somewhat mitigate this problem. The scheduled thread pool executor runs scheduled tasks in FIFO order so we shuffle the list of triggers before submitting them to the executor. I also added a cautionary note in the ref guide for the scheduled trigger describing this problem and recommending that scheduled triggers not be used for very frequent operations. > A trigger that creates trigger events more frequently than the cool down > period can starve other triggers > - > > Key: SOLR-13082 > URL: https://issues.apache.org/jira/browse/SOLR-13082 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.7 > > Attachments: SOLR-13082.patch > > > A trigger that creates frequent events (such as ScheduledTrigger) with period > less than the cool down period can starve all other triggers. This is because > we execute triggers in the same order each time so if the scheduled trigger > happens to be the first in the list then after every cool down period, the > scheduled trigger can create another event and the cycle repeats. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13683) SolrJ 8.1.1 Http2SolrClient should allow customizing HTTP headers
[
https://issues.apache.org/jira/browse/SOLR-13683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907760#comment-16907760
]
Shalin Shekhar Mangar commented on SOLR-13683:
--
bq. Our use case is indexing documents to Solr; hence, we do not use
SolrRequest APIs. We use SolrClient.add(collectionName,
Collection) API. Do you have any document which shows how to
use SolrRequest API for indexing use cases?
You can use UpdateRequest class which has an add method that accepts a
Collection instead of calling add on CloudHttp2SolrClient
directly.
> SolrJ 8.1.1 Http2SolrClient should allow customizing HTTP headers
> -
>
> Key: SOLR-13683
> URL: https://issues.apache.org/jira/browse/SOLR-13683
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: clients - java
>Affects Versions: 8.1.1
>Reporter: Niranjan Nanda
>Priority: Minor
>
> Currently {{Http2SolrClient}} does not allow configuring custom headers. For
> example, how to pass Basic Auth headers? It should expose some builder APIs
> to pass such headers.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
Shalin Shekhar Mangar created SOLR-13712: Summary: JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans Key: SOLR-13712 URL: https://issues.apache.org/jira/browse/SOLR-13712 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: JMX Affects Versions: 8.1.1, 8.2, 7.7.2, 6.6.5, 6.6.2, 6.6 Reporter: Shalin Shekhar Mangar Details to follow. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-13712:
-
Description:
This is quite easy to reproduce.
{code}
wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
tar xvf solr-6.6.0.tgz
cd solr-6.6.0
{code}
Enable jmx reporting by editing the server/solr/solr.xml and adding the
following under the "" tag:
{code}
{code}
Start solr with:
{code}
./bin/solr start -e cloud -noprompt
{code}
Open jconsole and inspect mbeans for solr nodes running on port 8983 and 7574.
You'll find that all mbeans (node, jvm, jetty, solr) are present for the solr
on port 8983 but completely absent for the solr node running on port 7574.
Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be fine.
was:Details to follow.
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913947#comment-16913947
]
Shalin Shekhar Mangar commented on SOLR-13712:
--
I tracked down the problem to SolrJmxReporter.
In the absence of any agentId or serviceUrl, the platform mbean server is
supposed to be used. The SolrJmxReporter.init (or doInit() method in later
versions) first tries to get the first mbean server and registers mbeans to it.
However, it exits early if no mbean server is found. Attaching a debugger to
the solr processes I found that the platform mbean server is always found for
the solr node running on port 8983 but not for solr on port 7574.
The problem is that the platform mbean server is created on the first
invocation of {{ManagementFactory.getPlatformMBeanServer()}} and
SolrJmxReporter calls this method in JmxMetricsReporter.build() which happens
after the mbean server is looked up. If no mbean server is found then the call
to build() never happens and therefore no mbeans are registered with the mbean
server.
So why does the mbean server exist on 8983? It is because of the embedded
zookeeper which calls {{ManagementFactory.getPlatformMBeanServer()}} before
solr tries to initialize SolrJmxReporter.
The code is virtually unchanged in Solr 7.x and 8.x. Then why do those versions
seem fine? It is because of log4j2 which registers its mbeans with the platform
mbean server before Solr initializes.
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13649) BasicAuth's 'blockUnknown' param should default to true
[
https://issues.apache.org/jira/browse/SOLR-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar reassigned SOLR-13649:
Assignee: Shalin Shekhar Mangar
> BasicAuth's 'blockUnknown' param should default to true
> ---
>
> Key: SOLR-13649
> URL: https://issues.apache.org/jira/browse/SOLR-13649
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, Authentication, security
>Affects Versions: 7.7.2, 8.1.1
> Environment: All
>Reporter: Marcus Eagan
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Labels: Authentication
> Fix For: master (9.0)
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> If someone seeks to enable basic authentication but they do not specify the
> {{blockUnknown}} parameter, the default value is {{false}}. That default
> behavior is a bit counterintuitive because if someone wishes to enable basic
> authentication, you would expect that they would want all unknown users to
> need to authenticate by default. I can imagine cases where you would not, but
> those cases would be less frequent.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-13712:
-
Attachment: SOLR-13712.patch
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-13712.patch
>
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar updated SOLR-13712:
-
Status: Patch Available (was: Open)
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-13712.patch
>
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915126#comment-16915126
]
Shalin Shekhar Mangar commented on SOLR-13712:
--
Patch for master which returns {{ManagementFactory.getPlatformMBeanServer()}}
inside JmxUtils.findFirstMBeanServer() if no mbean servers exist.
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-13712.patch
>
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13712) JMX MBeans are not exposed because of race condition between creating platform mbean server and registering mbeans
[
https://issues.apache.org/jira/browse/SOLR-13712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16915477#comment-16915477
]
Shalin Shekhar Mangar commented on SOLR-13712:
--
bq. It sounds like a workaround may be to configure log4j2 to register it's
mbeans earlier.
Yes, that should workaround this bug. I'll be interested to know what you find.
Thanks!
> JMX MBeans are not exposed because of race condition between creating
> platform mbean server and registering mbeans
> --
>
> Key: SOLR-13712
> URL: https://issues.apache.org/jira/browse/SOLR-13712
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: JMX
>Affects Versions: 6.6, 6.6.2, 6.6.5, 7.7.2, 8.2, 8.1.1
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-13712.patch
>
>
> This is quite easy to reproduce.
> {code}
> wget https://archive.apache.org/dist/lucene/solr/6.6.0/solr-6.6.0.tgz
> tar xvf solr-6.6.0.tgz
> cd solr-6.6.0
> {code}
> Enable jmx reporting by editing the server/solr/solr.xml and adding the
> following under the "" tag:
> {code}
>
> class="org.apache.solr.metrics.reporters.SolrJmxReporter" />
>
> {code}
> Start solr with:
> {code}
> ./bin/solr start -e cloud -noprompt
> {code}
> Open jconsole and inspect mbeans for solr nodes running on port 8983 and
> 7574. You'll find that all mbeans (node, jvm, jetty, solr) are present for
> the solr on port 8983 but completely absent for the solr node running on port
> 7574.
> Same behavior is on 6.6.2 and 6.6.6. However, Solr 7.x and 8.x seem to be
> fine.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13649) BasicAuth's 'blockUnknown' param should default to true
[
https://issues.apache.org/jira/browse/SOLR-13649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar resolved SOLR-13649.
--
Resolution: Fixed
Thanks Marcus and Jan for all the work!
> BasicAuth's 'blockUnknown' param should default to true
> ---
>
> Key: SOLR-13649
> URL: https://issues.apache.org/jira/browse/SOLR-13649
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI, Authentication, security
>Affects Versions: 7.7.2, 8.1.1
> Environment: All
>Reporter: Marcus Eagan
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Labels: Authentication
> Fix For: master (9.0)
>
> Time Spent: 9h 10m
> Remaining Estimate: 0h
>
> If someone seeks to enable basic authentication but they do not specify the
> {{blockUnknown}} parameter, the default value is {{false}}. That default
> behavior is a bit counterintuitive because if someone wishes to enable basic
> authentication, you would expect that they would want all unknown users to
> need to authenticate by default. I can imagine cases where you would not, but
> those cases would be less frequent.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12977) Autoscaling policy initialisation tries to fetch metrics from dead nodes
Shalin Shekhar Mangar created SOLR-12977:
Summary: Autoscaling policy initialisation tries to fetch metrics
from dead nodes
Key: SOLR-12977
URL: https://issues.apache.org/jira/browse/SOLR-12977
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: AutoScaling
Reporter: Shalin Shekhar Mangar
Fix For: 7.6, master (8.0)
Autoscaling policy initialisation tries to fetch metrics for each node during
construction. However, it does not skip the known dead nodes causing a timeout
to be logged. We should skip such requests entirely.
{code}
20579 WARN (AutoscalingActionExecutor-37-thread-1) []
o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node
127.0.0.1:63255_solr
org.apache.solr.client.solrj.SolrServerException: Server refused connection at:
http://127.0.0.1:63255/solr
at
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:650)
~[java/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
~[java/:?]
at
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
~[java/:?]
at
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) ~[java/:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
~[java/:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
[java/:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:186)
[java/:?]
at
org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getReplicaInfo(SolrClientNodeStateProvider.java:169)
[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Row.(Row.java:60) [java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:181)
[java/:?]
at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:114)
[java/:?]
at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:308)
[java/:?]
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12978) Autoscaling Suggester tries to test metrics for dead nodes
Shalin Shekhar Mangar created SOLR-12978:
Summary: Autoscaling Suggester tries to test metrics for dead nodes
Key: SOLR-12978
URL: https://issues.apache.org/jira/browse/SOLR-12978
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Components: AutoScaling, SolrCloud
Reporter: Shalin Shekhar Mangar
Fix For: 7.6, master (8.0)
Suggester tries to test clauses in the applyRules phase for each row regardless
of whether the row is live or not. When the node is not live and there are no
metrics fetched, testing the clause causes an NPE.
{code}
20586 WARN (AutoscalingActionExecutor-37-thread-1) []
o.a.s.c.a.ScheduledTriggers Exception executing actions
org.apache.solr.cloud.autoscaling.TriggerActionException: Error processing
action for trigger event: {
"id":"21d1e96fd8737T4ighk35ce6gv7f6h5zbndib4n",
"source":"node_lost_trigger",
"eventTime":594967172843319,
"eventType":"NODELOST",
"properties":{
"eventTimes":[594967172843319],
"preferredOperation":"movereplica",
"_enqueue_time_":594968181417909,
"nodeNames":["127.0.0.1:63255_solr"]}}
at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:311)
[java/:?]
at
org.apache.solr.cloud.autoscaling.ScheduledTriggers$$Lambda$498/1669229711.run(Unknown
Source) [java/:?]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[?:1.8.0_51]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_51]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$328(ExecutorUtil.java:209)
[java/:?]
at
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$10/1568754952.run(Unknown
Source) [java/:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_51]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_51]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_51]
Caused by: org.apache.solr.common.SolrException: Unexpected exception while
processing event: {
"id":"21d1e96fd8737T4ighk35ce6gv7f6h5zbndib4n",
"source":"node_lost_trigger",
"eventTime":594967172843319,
"eventType":"NODELOST",
"properties":{
"eventTimes":[594967172843319],
"preferredOperation":"movereplica",
"_enqueue_time_":594968181417909,
"nodeNames":["127.0.0.1:63255_solr"]}}
at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:160)
~[java/:?]
at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:308)
~[java/:?]
... 8 more
Caused by: java.lang.NullPointerException
at
org.apache.solr.client.solrj.cloud.autoscaling.RangeVal.match(RangeVal.java:34)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Operand$2.match(Operand.java:43)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Variable.match(Variable.java:46)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Variable$Type.match(Variable.java:358)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Condition.isPass(Condition.java:71)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Condition.isPass(Condition.java:76)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Clause.test(Clause.java:531)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Policy$Session.applyRules(Policy.java:635)
~[java/:?]
at
org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:185)
~[java/:?]
at
org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:114)
~[java/:?]
at
org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:308)
~[java/:?]
... 8 more
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16680527#comment-16680527
]
Shalin Shekhar Mangar commented on SOLR-12845:
--
Blocked by SOLR-12978
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Priority: Major
> Attachments: SOLR-12845.patch, SOLR-12845.patch
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7381) Improve Debuggability of SolrCloud using MDC
[ https://issues.apache.org/jira/browse/SOLR-7381?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16684166#comment-16684166 ] Shalin Shekhar Mangar commented on SOLR-7381: - [~yuzhih...@gmail.com] - No, our implementation explicitly sets MDC context from parent threads and unsets/restores old context once the submitted task finishes. See ExecutorUtil.MDCAwareThreadPoolExecutor for the implementation. > Improve Debuggability of SolrCloud using MDC > > > Key: SOLR-7381 > URL: https://issues.apache.org/jira/browse/SOLR-7381 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Critical > Fix For: 5.2, 6.0 > > Attachments: SOLR-7381-forbid-threadpoolexecutor.patch, > SOLR-7381-submitter-stacktrace.patch, SOLR-7381-thread-names.patch, > SOLR-7381-thread-names.patch, SOLR-7381-thread-names.patch, SOLR-7381.patch, > SOLR-7381.patch > > > SOLR-6673 added MDC based logging in a few places but we have a lot of ground > to cover. > # Threads created via thread pool executors do not inherit MDC values and > those are some of the most interesting places to log MDC context. > # We must expose node names (in tests) so that we can debug faster > # We can expose more information via thread names so that a thread dump has > enough context to help debug problems in production > This is critical to help debug SolrCloud failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12597) Migrate API should fail requests that do not specify split.key parameter
Shalin Shekhar Mangar created SOLR-12597: Summary: Migrate API should fail requests that do not specify split.key parameter Key: SOLR-12597 URL: https://issues.apache.org/jira/browse/SOLR-12597 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Affects Versions: 7.4 Reporter: Shalin Shekhar Mangar Fix For: master (8.0), 7.5 The migrate API should throw exception in case the split.key is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12597) Migrate API should fail requests that do not specify split.key parameter
[ https://issues.apache.org/jira/browse/SOLR-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-12597: - Attachment: SOLR-12597.patch > Migrate API should fail requests that do not specify split.key parameter > > > Key: SOLR-12597 > URL: https://issues.apache.org/jira/browse/SOLR-12597 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 7.4 >Reporter: Shalin Shekhar Mangar >Priority: Trivial > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12597.patch > > > The migrate API should throw exception in case the split.key is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-12597) Migrate API should fail requests that do not specify split.key parameter
[ https://issues.apache.org/jira/browse/SOLR-12597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar resolved SOLR-12597. -- Resolution: Fixed Assignee: Shalin Shekhar Mangar > Migrate API should fail requests that do not specify split.key parameter > > > Key: SOLR-12597 > URL: https://issues.apache.org/jira/browse/SOLR-12597 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 7.4 >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Trivial > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12597.patch > > > The migrate API should throw exception in case the split.key is missing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-11990) Make it possible to co-locate replicas of multiple collections together in a node
[
https://issues.apache.org/jira/browse/SOLR-11990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shalin Shekhar Mangar resolved SOLR-11990.
--
Resolution: Fixed
Thanks [~noble.paul] for the help in designing and reviewing this piece.
> Make it possible to co-locate replicas of multiple collections together in a
> node
> -
>
> Key: SOLR-11990
> URL: https://issues.apache.org/jira/browse/SOLR-11990
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch,
> SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch, SOLR-11990.patch
>
>
> It is necessary to co-locate replicas of different collection together in a
> node when cross-collection joins are performed.
> while creating a collection specify the parameter
> {{withCollection=other-collection-name}} . This ensure that Solr always
> ensure that atleast one replica of {{other-collection}} is present with this
> collection replicas
> This requires changing create collection, create shard and add replica APIs
> as well because we want a replica of collection A to be created first before
> a replica of collection B is created so that join queries etc are always
> possible.
> Some caveats to this implementation:
> # The {{other-collection}} should only have a single shard named "shard1"
> # Any replica of {{other-collection}} created by this feature will be of NRT
> type
> Removing the above caveats can be a goal of other issues.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12607) Investigate ShardSplitTest failures
Shalin Shekhar Mangar created SOLR-12607:
Summary: Investigate ShardSplitTest failures
Key: SOLR-12607
URL: https://issues.apache.org/jira/browse/SOLR-12607
Project: Solr
Issue Type: Task
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Reporter: Shalin Shekhar Mangar
Assignee: Shalin Shekhar Mangar
Fix For: master (8.0), 7.5
There have been many recent ShardSplitTest failures.
According to http://fucit.org/solr-jenkins-reports/failure-report.html
{code}
Class: org.apache.solr.cloud.api.collections.ShardSplitTest
Method: testSplitWithChaosMonkey
Failures: 72.32% (81 / 112)
Class: org.apache.solr.cloud.api.collections.ShardSplitTest
Method: test
Failures: 26.79% (30 / 112)
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563230#comment-16563230
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
The testSplitWithChaosMonkey failures increased noticeably after SOLR-11665 was
committed. I looked at the logs of a recent failure and here's what I found:
# Shard Split succeeds in creating new sub-shards and new replicas
# The leader node is killed by chaos monkey before the new replicas can become
active
# SOLR-11665 kicks in and cleans up (deletes) the sub-shards in construction
including all their state from ZK
# The old leader node is started up again and re-registers the local cores
thereby creating state in ZK again. However this time, since the parent shard
information was deleted by the cleanup, the state is missing parent and range
and slice state is set to active.
# This causes the assertions in the test to fail i.e. either no sub-shards
exist or if they do, they are active and recovered
There are two bugs in play here:
# The async API status of the split shard command is COMPLETED instead of
FAILED which leads the test to believe that the sub-shard slice and replicas
should exist but they don't.
# By default, our tests still use legacyCloud=true unless set otherwise.
I'll set legacyCloud=false for this test and open another issue to set this to
false by default throughout the test suite.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563230#comment-16563230
]
Shalin Shekhar Mangar edited comment on SOLR-12607 at 7/31/18 8:30 AM:
---
The testSplitWithChaosMonkey failures increased noticeably after SOLR-11665 was
committed. I looked at the logs of a recent failure and here's what I found:
# Shard Split succeeds in creating new sub-shards and new replicas
# The leader node is killed by chaos monkey before the new replicas can become
active
# SOLR-11665 kicks in and cleans up (deletes) the sub-shards in construction
including all their state from ZK
# The old leader node is started up again and re-registers the local cores
thereby creating state in ZK again. However this time, since the parent shard
information was deleted by the cleanup, the state is missing parent and range
and slice state is set to active.
# This causes the assertions in the test to fail i.e. either no sub-shards
exist or if they do, they are active and recovered
There are two bugs in play here:
# -The async API status of the split shard command is COMPLETED instead of
FAILED which leads the test to believe that the sub-shard slice and replicas
should exist but they don't.-
# By default, our tests still use legacyCloud=true unless set otherwise.
I'll set legacyCloud=false for this test and open another issue to set this to
false by default throughout the test suite.
was (Author: shalinmangar):
The testSplitWithChaosMonkey failures increased noticeably after SOLR-11665 was
committed. I looked at the logs of a recent failure and here's what I found:
# Shard Split succeeds in creating new sub-shards and new replicas
# The leader node is killed by chaos monkey before the new replicas can become
active
# SOLR-11665 kicks in and cleans up (deletes) the sub-shards in construction
including all their state from ZK
# The old leader node is started up again and re-registers the local cores
thereby creating state in ZK again. However this time, since the parent shard
information was deleted by the cleanup, the state is missing parent and range
and slice state is set to active.
# This causes the assertions in the test to fail i.e. either no sub-shards
exist or if they do, they are active and recovered
There are two bugs in play here:
# The async API status of the split shard command is COMPLETED instead of
FAILED which leads the test to believe that the sub-shard slice and replicas
should exist but they don't.
# By default, our tests still use legacyCloud=true unless set otherwise.
I'll set legacyCloud=false for this test and open another issue to set this to
false by default throughout the test suite.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563298#comment-16563298
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
I was wrong about the status being reported incorrectly. Actually the test was
logging the status as COMPLETE instead of the actual status.
I've pushed fixes and removed the badapple annotation to the branch
{{jira/solr-12607}}. I am beasting this test now and will commit once tests
finish successfully.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563453#comment-16563453
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
Beasting 200 rounds finished successfully. Now on to failures of the {{test()}}
method.
Looking at the failures logs from various jenkins machines as well as locally,
I see that the failures are isolated to runs where the replica types are TLOG.
I'm going to hard code replica types to NRT and beast the test to see if it
passes.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563482#comment-16563482
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
Beasting 100 rounds with replica types fixed to NRT finished successfully. I'm
debugging the test with TLOG replica types to see what's going wrong.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563530#comment-16563530
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
It looks like that when documents are forwarded from parent shard leader to the
sub-shard leader then the sub-shard leader being of type TLOG and not executing
leader logic, adds the IGNORE_INDEXWRITER flag to the command causing the
document to not be indexed. Still digging.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12574) SignificantTermsQParserPlugin should output its keys in a combined bucket
[
https://issues.apache.org/jira/browse/SOLR-12574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564904#comment-16564904
]
Shalin Shekhar Mangar commented on SOLR-12574:
--
[~arafalov] -- this needs backporting to 7x too.
> SignificantTermsQParserPlugin should output its keys in a combined bucket
> -
>
> Key: SOLR-12574
> URL: https://issues.apache.org/jira/browse/SOLR-12574
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: 7.4
>Reporter: Alexandre Rafalovitch
>Assignee: Alexandre Rafalovitch
>Priority: Minor
> Fix For: 7.5
>
> Attachments: SOLR-12574.patch
>
>
> SignificantTermsQParserPlugin is not yet visible to the users (was not
> documented or spelt correctly in 7.4), so there is still a chance to fix its
> output before people start using it.
> Currently, it injects 6 different keys into the document, on the same level
> as responseHeader and response. This feels like polluting top-level space. It
> may be better to put all those keys under one bucket (e.g. significantTerms).
> Additionally, resultCount is always the same as response.numFound (documents
> found), so does not seem to be needed.
> Current output:
> {code:java}
> {
> "responseHeader": {
> "status": 0,
> "QTime": 1,
> "params": {
> "q": "directed_by_str:\"Steven Soderbergh\"",
> "fq": "{!significantTerms field=genre numTerms=2}",
> "rows": "1",
> "wt": "json"
> }
> },
> "numDocs": 1100,
> "resultCount": 5,
> "sterms": [
> "biographical",
> "romance"
> ],
> "scores": [
> 2.5552773475646973,
> 2.6387078762054443
> ],
> "docFreq": [
> 74,
> 270
> ],
> "queryDocFreq": [
> 2,
> 3
> ],
> "response": {
> "numFound": 5,
> "start": 0,
> "docs": [
> {
> "id": "/en/bubble",
> "directed_by": [
> "Steven Soderbergh"
> ],
> "initial_release_date": "2005-09-03T00:00:00Z",
> "name": "Bubble",
> "genre": [
> "Crime Fiction",
> "Mystery",
> "Indie film",
> "Thriller",
> "Drama"
> ],
> "_version_": 1606610059993808899
> }
> ]
> }
> }{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16564939#comment-16564939
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
Thanks [~erickerickson]. I have a couple nodes to test myself but I'll let you
know if I need more help.
[~caomanhdat] helped track down the problem. A isSubShardLeader check was
missing which caused this problem. I've pushed updates to the branch and I'm
beasting the tests with only tlog replicas enabled.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565081#comment-16565081
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
Beasting passed for the tlog replicas but it uncovered yet another failure
(unrelated to tlog replicas) that happens once in 200 runs approximately.
The DistributedUpdateProcessor's doFinish() method relies on the failing node's
shard name to decide whether to return an exception to the client or to
continue. During shard split, we forward updates synchronously to sub shard
leaders and we'd like to return exceptions to clients so that they can retry.
However, due to using the parent shard name in the StdNode created for
sub-shard leader, the comparison between the current replica's shard and
StdNode's shard always passed and the errors were never returned. I've
committed a fix to the branch and I'll add a test to cover this particular
situation.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-12610) Inject failures during synchronous update requests during shard splits
Shalin Shekhar Mangar created SOLR-12610: Summary: Inject failures during synchronous update requests during shard splits Key: SOLR-12610 URL: https://issues.apache.org/jira/browse/SOLR-12610 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Reporter: Shalin Shekhar Mangar Assignee: Shalin Shekhar Mangar Fix For: master (8.0), 7.5 In SOLR-12607, I found a bug where the StdNode's shard was not set correctly causing exceptions during updates forwarded to sub-shard leaders to not be sent back to the clients. This can cause data loss during split. A fix was committed as part of SOLR-12607 but we need to expand coverage to this situation. I'll add failure injection during the synchronous update step to simulate this condition. This will be randomized for each shard split test method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12610) Inject failures during synchronous update requests during shard splits
[ https://issues.apache.org/jira/browse/SOLR-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-12610: - Attachment: SOLR-12610.patch > Inject failures during synchronous update requests during shard splits > -- > > Key: SOLR-12610 > URL: https://issues.apache.org/jira/browse/SOLR-12610 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12610.patch > > > In SOLR-12607, I found a bug where the StdNode's shard was not set correctly > causing exceptions during updates forwarded to sub-shard leaders to not be > sent back to the clients. This can cause data loss during split. A fix was > committed as part of SOLR-12607 but we need to expand coverage to this > situation. I'll add failure injection during the synchronous update step to > simulate this condition. This will be randomized for each shard split test > method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12610) Inject failures during synchronous update requests during shard splits
[ https://issues.apache.org/jira/browse/SOLR-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565168#comment-16565168 ] Shalin Shekhar Mangar commented on SOLR-12610: -- There are some failures with this randomization enabled. I see requests being retried by CloudSolrClient when there is no communication exception and no stale state. > Inject failures during synchronous update requests during shard splits > -- > > Key: SOLR-12610 > URL: https://issues.apache.org/jira/browse/SOLR-12610 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12610.patch > > > In SOLR-12607, I found a bug where the StdNode's shard was not set correctly > causing exceptions during updates forwarded to sub-shard leaders to not be > sent back to the clients. This can cause data loss during split. A fix was > committed as part of SOLR-12607 but we need to expand coverage to this > situation. I'll add failure injection during the synchronous update step to > simulate this condition. This will be randomized for each shard split test > method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565180#comment-16565180
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
Beasting passed with the fix I committed. I've opened SOLR-12610 to inject
failures during the synchronous forwarding step to cover this scenario.
I beasted the third failing test for this class testSplitMixedReplicaTypes a
couple 100 times. I peeked at some of the past failures and they were also due
to issues in handling tlog replica types which I fixed already.
I'll merge this to master but keep the issue open for a day or two.
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12607) Investigate ShardSplitTest failures
[
https://issues.apache.org/jira/browse/SOLR-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16565519#comment-16565519
]
Shalin Shekhar Mangar commented on SOLR-12607:
--
[~erickerickson] - I'm going to merge this tomorrow my time so if you have free
resources, beast away on ShardSplitTest on branch {{jira/solr-12607}} and let
me know how it looks. Thanks!
> Investigate ShardSplitTest failures
> ---
>
> Key: SOLR-12607
> URL: https://issues.apache.org/jira/browse/SOLR-12607
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
>
> There have been many recent ShardSplitTest failures.
> According to http://fucit.org/solr-jenkins-reports/failure-report.html
> {code}
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: testSplitWithChaosMonkey
> Failures: 72.32% (81 / 112)
> Class: org.apache.solr.cloud.api.collections.ShardSplitTest
> Method: test
> Failures: 26.79% (30 / 112)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12610) Inject failures during synchronous update requests during shard splits
[ https://issues.apache.org/jira/browse/SOLR-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-12610: - Attachment: SOLR-12610-test-cloudclient-retry.patch > Inject failures during synchronous update requests during shard splits > -- > > Key: SOLR-12610 > URL: https://issues.apache.org/jira/browse/SOLR-12610 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: Shalin Shekhar Mangar >Assignee: Shalin Shekhar Mangar >Priority: Major > Fix For: master (8.0), 7.5 > > Attachments: SOLR-12610-test-cloudclient-retry.patch, SOLR-12610.patch > > > In SOLR-12607, I found a bug where the StdNode's shard was not set correctly > causing exceptions during updates forwarded to sub-shard leaders to not be > sent back to the clients. This can cause data loss during split. A fix was > committed as part of SOLR-12607 but we need to expand coverage to this > situation. I'll add failure injection during the synchronous update step to > simulate this condition. This will be randomized for each shard split test > method. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12610) Inject failures during synchronous update requests during shard splits
[
https://issues.apache.org/jira/browse/SOLR-12610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16566452#comment-16566452
]
Shalin Shekhar Mangar commented on SOLR-12610:
--
The new {{SOLR-12610-test-cloudclient-retry.patch}} has a test which tests
retry behavior of CloudSolrClient. The HTTP 500 error from the server is not a
communication problem and the cluster state is also unchanged and yet the
CloudSolrClient attempts retries.
> Inject failures during synchronous update requests during shard splits
> --
>
> Key: SOLR-12610
> URL: https://issues.apache.org/jira/browse/SOLR-12610
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Shalin Shekhar Mangar
>Priority: Major
> Fix For: master (8.0), 7.5
>
> Attachments: SOLR-12610-test-cloudclient-retry.patch, SOLR-12610.patch
>
>
> In SOLR-12607, I found a bug where the StdNode's shard was not set correctly
> causing exceptions during updates forwarded to sub-shard leaders to not be
> sent back to the clients. This can cause data loss during split. A fix was
> committed as part of SOLR-12607 but we need to expand coverage to this
> situation. I'll add failure injection during the synchronous update step to
> simulate this condition. This will be randomized for each shard split test
> method.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12130) Investigate why CdcrReplicationDistributedZkTest is slow
[ https://issues.apache.org/jira/browse/SOLR-12130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570392#comment-16570392 ] Shalin Shekhar Mangar commented on SOLR-12130: -- What's the status on these improvements? Is it just waiting for a review+commit? > Investigate why CdcrReplicationDistributedZkTest is slow > > > Key: SOLR-12130 > URL: https://issues.apache.org/jira/browse/SOLR-12130 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Assignee: Varun Thacker >Priority: Major > Attachments: SOLR-12130.patch, SOLR-12130.patch, SOLR-12130.patch > > > CdcrReplicationDistributedZkTest seems to be a very slow test and probably > why it was marked nightly in the first place? > Investigate why the test is so slow and see if we can speed it up -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-10032) Create report to assess Solr test quality at a commit point.
[ https://issues.apache.org/jira/browse/SOLR-10032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16570398#comment-16570398 ] Shalin Shekhar Mangar commented on SOLR-10032: -- Hi [~markrmil...@gmail.com], the last report was in April. Is this still something you intend to publish and/or publish the code? > Create report to assess Solr test quality at a commit point. > > > Key: SOLR-10032 > URL: https://issues.apache.org/jira/browse/SOLR-10032 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Fix For: 7.0, master (8.0) > > Attachments: Lucene-Solr Master Test Beast Results > 01-24-2017-9899cbd031dc3fc37a384b1f9e2b379e90a9a3a6 Level Medium- Running 30 > iterations, 12 at a time .pdf, Lucene-Solr Master Test Beasults > 02%2F17%2F2017-19c8ec2bf1882bed1bb34d0b55198d03f2018838 Level Hard Running > 100 iterations, 12 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults > 02-01-2017-bbc455de195c83d9f807980b510fa46018f33b1b Level Medium- Running 30 > iterations, 10 at a time.pdf, Lucene-Solr Master Test Beasults > 02-08-2017-6696eafaae18948c2891ce758c7a2ec09873dab8 Level Medium+- Running 30 > iterations, 10 at a time, 8 cores.pdf, Lucene-Solr Master Test Beasults > 02-14-2017- Level Medium+-a1f114f70f3800292c25be08213edf39b3e37f6a Running 30 > iterations, 10 at a time, 8 cores.pdf > > > We have many Jenkins instances blasting tests, some official, some policeman, > I and others have or had their own, and the email trail proves the power of > the Jenkins cluster to find test fails. > However, I still have a very hard time with some basic questions: > what tests are flakey right now? which test fails actually affect devs most? > did I break it? was that test already flakey? is that test still flakey? what > are our worst tests right now? is that test getting better or worse? > We really need a way to see exactly what tests are the problem, not because > of OS or environmental issues, but more basic test quality issues. Which > tests are flakey and how flakey are they at any point in time. > Reports: > https://drive.google.com/drive/folders/0ByYyjsrbz7-qa2dOaU1UZDdRVzg?usp=sharing > 01/24/2017 - > https://docs.google.com/spreadsheets/d/1JySta2j2s7A_p16wA1UO-l6c4GsUHBIb4FONS2EzW9k/edit?usp=sharing > 02/01/2017 - > https://docs.google.com/spreadsheets/d/1FndoyHmihaOVL2o_Zns5alpNdAJlNsEwQVoJ4XDWj3c/edit?usp=sharing > 02/08/2017 - > https://docs.google.com/spreadsheets/d/1N6RxH4Edd7ldRIaVfin0si-uSLGyowQi8-7mcux27S0/edit?usp=sharing > 02/14/2017 - > https://docs.google.com/spreadsheets/d/1eZ9_ds_0XyqsKKp8xkmESrcMZRP85jTxSKkNwgtcUn0/edit?usp=sharing > 02/17/2017 - > https://docs.google.com/spreadsheets/d/1LEPvXbsoHtKfIcZCJZ3_P6OHp7S5g2HP2OJgU6B2sAg/edit?usp=sharing -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-12739) Make autoscaling policy based replica placement the default strategy for placing replicas
[ https://issues.apache.org/jira/browse/SOLR-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-12739: - Attachment: SOLR-12739.patch > Make autoscaling policy based replica placement the default strategy for > placing replicas > - > > Key: SOLR-12739 > URL: https://issues.apache.org/jira/browse/SOLR-12739 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-12739.patch, SOLR-12739.patch, SOLR-12739.patch, > SOLR-12739.patch > > > Today the default placement strategy is the same one used since Solr 4.x > which is to select nodes on a round robin fashion. I propose to make the > autoscaling policy based replica placement as the default policy for placing > replicas. > This is related to SOLR-12648 where even though we have default cluster > preferences, we don't use them unless a policy is also configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12739) Make autoscaling policy based replica placement the default strategy for placing replicas
[ https://issues.apache.org/jira/browse/SOLR-12739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16641758#comment-16641758 ] Shalin Shekhar Mangar commented on SOLR-12739: -- This patch fixes various autoscaling tests that never had to specify a maxShardsPerNode before but must now. Not only did Solr not support maxShardsPerNode when autoscaling was enabled, it set maxShardsPerNode as unlimited (Integer.MAX_VALUE) so that maxShardsPerNode never interferes with autoscaling policy. I'm still undecided whether to continue the old behavior of unlimited maxShardsPerNode or not. Only one test failures remains: LeaderVoteWaitTimeoutTest. > Make autoscaling policy based replica placement the default strategy for > placing replicas > - > > Key: SOLR-12739 > URL: https://issues.apache.org/jira/browse/SOLR-12739 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: AutoScaling, SolrCloud >Reporter: Shalin Shekhar Mangar >Priority: Major > Fix For: 7.6, master (8.0) > > Attachments: SOLR-12739.patch, SOLR-12739.patch, SOLR-12739.patch, > SOLR-12739.patch > > > Today the default placement strategy is the same one used since Solr 4.x > which is to select nodes on a round robin fashion. I propose to make the > autoscaling policy based replica placement as the default policy for placing > replicas. > This is related to SOLR-12648 where even though we have default cluster > preferences, we don't use them unless a policy is also configured. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
