[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3705 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3705/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth Error Message: expected:<200> but was:<403> Stack Trace: java.lang.AssertionError: expected:<200> but was:<403> at __randomizedtesting.SeedInfo.seed([D4092FA9C13739D3:686759BB6564BAA9]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.junit.Assert.assertEquals(Assert.java:456) at org.apache.solr.security.BasicAuthIntegrationTest.executeCommand(BasicAuthIntegrationTest.java:231) at org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth(BasicAuthIntegrationTest.java:143) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang.Thread.run(Thread.java:745)
[jira] [Closed] (SOLR-9846) OverseerAutoReplicaFailoverThread can take too long to stop and leak out of unit tests.
[ https://issues.apache.org/jira/browse/SOLR-9846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller closed SOLR-9846. - Resolution: Duplicate > OverseerAutoReplicaFailoverThread can take too long to stop and leak out of > unit tests. > --- > > Key: SOLR-9846 > URL: https://issues.apache.org/jira/browse/SOLR-9846 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller > > We should interrupt it on close. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-9850) Break the mega test SplitShardTest into multiple tests.
[ https://issues.apache.org/jira/browse/SOLR-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller reassigned SOLR-9850: - Assignee: Mark Miller > Break the mega test SplitShardTest into multiple tests. > --- > > Key: SOLR-9850 > URL: https://issues.apache.org/jira/browse/SOLR-9850 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-9850.patch > > > This mega test is out of control in test length and needs to be broken up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9850) Break the mega test SplitShardTest into multiple tests.
[ https://issues.apache.org/jira/browse/SOLR-9850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-9850: -- Attachment: SOLR-9850.patch > Break the mega test SplitShardTest into multiple tests. > --- > > Key: SOLR-9850 > URL: https://issues.apache.org/jira/browse/SOLR-9850 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller > Attachments: SOLR-9850.patch > > > This mega test is out of control in test length and needs to be broken up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9850) Break the mega test SplitShardTest into multiple tests.
Mark Miller created SOLR-9850: - Summary: Break the mega test SplitShardTest into multiple tests. Key: SOLR-9850 URL: https://issues.apache.org/jira/browse/SOLR-9850 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller This mega test is out of control in test length and needs to be broken up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7333) Make the poll queue time configurable and use knowledge that a batch is being processed to poll efficiently
[
https://issues.apache.org/jira/browse/SOLR-7333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740962#comment-15740962
]
Mark Miller commented on SOLR-7333:
---
We are trying to improve on this in SOLR-9824 so that we use minimal requests
and also avoid needless waits after updates for all cases.
This only worked if you used javabin and it only worked for batched docs in a
request - streaming saw no benefit.
> Make the poll queue time configurable and use knowledge that a batch is being
> processed to poll efficiently
> ---
>
> Key: SOLR-7333
> URL: https://issues.apache.org/jira/browse/SOLR-7333
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Reporter: Timothy Potter
>Assignee: Timothy Potter
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7333.patch, SOLR-7333.patch
>
>
> {{StreamingSolrClients}} uses {{ConcurrentUpdateSolrServer}} to stream
> documents from leader to replica, by default it sets the {{pollQueueTime}}
> for CUSS to 0 so that we don't impose an unnecessary wait when processing
> single document updates or the last doc in a batch. However, the downside is
> that replicas receive many more update requests than leaders; I've seen up to
> 40x number of update requests between replica and leader.
> If we're processing a batch of docs, then ideally the poll queue time should
> be greater than 0 up until the last doc is pulled off the queue. If we're
> processing a single doc, then the poll queue time should always be 0 as we
> don't want the thread to wait unnecessarily for another doc that won't come.
> Rather than force indexing applications to provide this optional parameter in
> an update request, it would be better for server-side code that can detect
> whether an update request is a single document or batch of documents to
> override this value internally, i.e. it'll be 0 by default, but since
> {{JavaBinUpdateRequestCodec}} can determine when it's seen the last doc in a
> batch, it can override the pollQueueTime to something greater than 0.
> This means that current indexing clients will see a boost when doing batch
> updates without making any changes on their side.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-master - Build # 1537 - Unstable
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1537/ 1 tests failed. FAILED: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored Error Message: There are still nodes recoverying - waited for 330 seconds Stack Trace: java.lang.AssertionError: There are still nodes recoverying - waited for 330 seconds at __randomizedtesting.SeedInfo.seed([C3613E98031A38D1:515D5F1CBF5D0509]:0) at org.junit.Assert.fail(Assert.java:93) at org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:184) at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.waitForRecoveriesToFinish(TestStressCloudBlindAtomicUpdates.java:459) at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.checkField(TestStressCloudBlindAtomicUpdates.java:304) at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates.test_dv_stored(TestStressCloudBlindAtomicUpdates.java:203) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Created] (SOLR-9849) Use a very low value for solr.cloud.wait-for-updates-with-stale-state-pause in tests.
Mark Miller created SOLR-9849: - Summary: Use a very low value for solr.cloud.wait-for-updates-with-stale-state-pause in tests. Key: SOLR-9849 URL: https://issues.apache.org/jira/browse/SOLR-9849 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller Assignee: Mark Miller -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_112) - Build # 6282 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6282/ Java: 32bit/jdk1.8.0_112 -server -XX:+UseParallelGC 1 tests failed. FAILED: org.apache.solr.handler.TestReplicationHandler.doTestStressReplication Error Message: [index.20161212024128374, index.20161212024140345, index.properties, replication.properties, snapshot_metadata] expected:<1> but was:<2> Stack Trace: java.lang.AssertionError: [index.20161212024128374, index.20161212024140345, index.properties, replication.properties, snapshot_metadata] expected:<1> but was:<2> at __randomizedtesting.SeedInfo.seed([88B9F54D6A112293:5312F58B6F394B20]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.apache.solr.handler.TestReplicationHandler.checkForSingleIndex(TestReplicationHandler.java:907) at org.apache.solr.handler.TestReplicationHandler.doTestStressReplication(TestReplicationHandler.java:874) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at
[jira] [Created] (SOLR-9848) Lower solr.cloud.wait-for-updates-with-stale-state-pause back down from 7 seconds.
Mark Miller created SOLR-9848: - Summary: Lower solr.cloud.wait-for-updates-with-stale-state-pause back down from 7 seconds. Key: SOLR-9848 URL: https://issues.apache.org/jira/browse/SOLR-9848 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller While looking for replica out of sync issues we jacked this up hoping it might be involved, but I don't that it was and we should look at returning to a 2 second wait or so by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4587) Implement Saved Searches a la ElasticSearch Percolator
[ https://issues.apache.org/jira/browse/SOLR-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740831#comment-15740831 ] Hemant Purswani commented on SOLR-4587: --- Seems like topic() function is still in Beta (https://issues.apache.org/jira/browse/SOLR-8709) > Implement Saved Searches a la ElasticSearch Percolator > -- > > Key: SOLR-4587 > URL: https://issues.apache.org/jira/browse/SOLR-4587 > Project: Solr > Issue Type: New Feature > Components: SearchComponents - other, SolrCloud >Reporter: Otis Gospodnetic > Fix For: 6.0 > > > Use Lucene MemoryIndex for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4587) Implement Saved Searches a la ElasticSearch Percolator
[ https://issues.apache.org/jira/browse/SOLR-4587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740825#comment-15740825 ] Hemant Purswani commented on SOLR-4587: --- Is there a working example of using topic() function for alerts. Is the stream API robust enough to be used in production? > Implement Saved Searches a la ElasticSearch Percolator > -- > > Key: SOLR-4587 > URL: https://issues.apache.org/jira/browse/SOLR-4587 > Project: Solr > Issue Type: New Feature > Components: SearchComponents - other, SolrCloud >Reporter: Otis Gospodnetic > Fix For: 6.0 > > > Use Lucene MemoryIndex for this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9847) Deadlock on ManagedIndexSchema lock.
[
https://issues.apache.org/jira/browse/SOLR-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740813#comment-15740813
]
Mark Miller commented on SOLR-9847:
---
The stacks:
{noformat}
qtp1112580055-3894 [WAITING]
sun.misc.Unsafe.park(boolean, long) Unsafe.java (native)
java.util.concurrent.locks.LockSupport.parkNanos(Object, long)
LockSupport.java:215
java.util.concurrent.FutureTask.awaitDone(boolean, long) FutureTask.java:426
java.util.concurrent.FutureTask.get(long, TimeUnit) FutureTask.java:204
java.util.concurrent.AbstractExecutorService.invokeAll(Collection, long,
TimeUnit) AbstractExecutorService.java:289
org.apache.solr.schema.ManagedIndexSchema.waitForSchemaZkVersionAgreement(String,
String, int, ZkController, int) ManagedIndexSchema.java:238
org.apache.solr.schema.SchemaManager.waitForOtherReplicasToUpdate(TimeOut, int)
SchemaManager.java:167
org.apache.solr.schema.SchemaManager.doOperations(List) SchemaManager.java:137
org.apache.solr.schema.SchemaManager.performOperations(Reader)
SchemaManager.java:92
org.apache.solr.handler.SchemaHandler.handleRequestBody(SolrQueryRequest,
SolrQueryResponse) SchemaHandler.java:91
org.apache.solr.handler.RequestHandlerBase.handleRequest(SolrQueryRequest,
SolrQueryResponse) RequestHandlerBase.java:152
org.apache.solr.core.SolrCore.execute(SolrRequestHandler, SolrQueryRequest,
SolrQueryResponse) SolrCore.java:2227
{noformat}
{noformat}
zkCallback-535-thread-2-processing-n:127.0.0.1:34881_c_c [BLOCKED]
org.apache.solr.schema.ZkIndexSchemaReader.updateSchema(Watcher, int)
ZkIndexSchemaReader.java:131
org.apache.solr.schema.ZkIndexSchemaReader.access$200(ZkIndexSchemaReader,
Watcher, int) ZkIndexSchemaReader.java:39
org.apache.solr.schema.ZkIndexSchemaReader$2.process(WatchedEvent)
ZkIndexSchemaReader.java:97
org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(Watcher,
WatchedEvent) SolrZkClient.java:268
org.apache.solr.common.cloud.SolrZkClient$3$$Lambda$62.run()
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ArrayList,
List, Map, String, Runnable, Exception) ExecutorUtil.java:229
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$40.run()
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{noformat}
{noformat}
zkCallback-522-thread-2-processing-n:127.0.0.1:39199_c_c [BLOCKED]
org.apache.solr.schema.ZkIndexSchemaReader.updateSchema(Watcher, int)
ZkIndexSchemaReader.java:131
org.apache.solr.schema.ZkIndexSchemaReader.access$200(ZkIndexSchemaReader,
Watcher, int) ZkIndexSchemaReader.java:39
org.apache.solr.schema.ZkIndexSchemaReader$2.process(WatchedEvent)
ZkIndexSchemaReader.java:97
org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$0(Watcher,
WatchedEvent) SolrZkClient.java:268
org.apache.solr.common.cloud.SolrZkClient$3$$Lambda$62.run()
java.util.concurrent.Executors$RunnableAdapter.call() Executors.java:511
java.util.concurrent.FutureTask.run() FutureTask.java:266
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ArrayList,
List, Map, String, Runnable, Exception) ExecutorUtil.java:229
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$40.run()
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor$Worker)
ThreadPoolExecutor.java:1142
java.util.concurrent.ThreadPoolExecutor$Worker.run() ThreadPoolExecutor.java:617
java.lang.Thread.run() Thread.java:745
{noformat}
> Deadlock on ManagedIndexSchema lock.
>
>
> Key: SOLR-9847
> URL: https://issues.apache.org/jira/browse/SOLR-9847
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mark Miller
> Attachments: SOLR-9847.patch
>
>
> Seems we hold the lock while in
> ManagedIndexSchema.waitForSchemaZkVersionAgreement, so we may never see
> agreement because are updates may also be waiting on that lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9847) Deadlock on ManagedIndexSchema lock.
[ https://issues.apache.org/jira/browse/SOLR-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-9847: -- Attachment: SOLR-9847.patch I don't know this code so well, but here is one idea. > Deadlock on ManagedIndexSchema lock. > > > Key: SOLR-9847 > URL: https://issues.apache.org/jira/browse/SOLR-9847 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller > Attachments: SOLR-9847.patch > > > Seems we hold the lock while in > ManagedIndexSchema.waitForSchemaZkVersionAgreement, so we may never see > agreement because are updates may also be waiting on that lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9847) Deadlock on ManagedIndexSchema lock.
[ https://issues.apache.org/jira/browse/SOLR-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740801#comment-15740801 ] Mark Miller commented on SOLR-9847: --- I suppose that eventually happens because of: // eventually, this loop will get killed by the ExecutorService's timeout > Deadlock on ManagedIndexSchema lock. > > > Key: SOLR-9847 > URL: https://issues.apache.org/jira/browse/SOLR-9847 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller > > Seems we hold the lock while in > ManagedIndexSchema.waitForSchemaZkVersionAgreement, so we may never see > agreement because are updates may also be waiting on that lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-9847) Deadlock on ManagedIndexSchema lock.
[
https://issues.apache.org/jira/browse/SOLR-9847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740799#comment-15740799
]
Mark Miller commented on SOLR-9847:
---
Not a complete deadlock I guess, but takes a long time to sort out:
{noformat}
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:27:14, stalled for
190s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:28:14, stalled for
250s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:29:14, stalled for
310s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:30:14, stalled for
370s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:31:14, stalled for
430s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:32:14, stalled for
490s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:33:14, stalled for
550s at: TestBulkSchemaConcurrent.test
[junit4] HEARTBEAT J9 PID(4929@totalmetal): 2016-12-11T21:34:14, stalled for
610s at: TestBulkSchemaConcurrent.test
[junit4] Suite: org.apache.solr.schema.TestBulkSchemaConcurrent
{noformat}
> Deadlock on ManagedIndexSchema lock.
>
>
> Key: SOLR-9847
> URL: https://issues.apache.org/jira/browse/SOLR-9847
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mark Miller
>
> Seems we hold the lock while in
> ManagedIndexSchema.waitForSchemaZkVersionAgreement, so we may never see
> agreement because are updates may also be waiting on that lock.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9847) Deadlock on ManagedIndexSchema lock.
Mark Miller created SOLR-9847: - Summary: Deadlock on ManagedIndexSchema lock. Key: SOLR-9847 URL: https://issues.apache.org/jira/browse/SOLR-9847 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller Seems we hold the lock while in ManagedIndexSchema.waitForSchemaZkVersionAgreement, so we may never see agreement because are updates may also be waiting on that lock. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9846) OverseerAutoReplicaFailoverThread can take too long to stop and leak out of unit tests.
Mark Miller created SOLR-9846: - Summary: OverseerAutoReplicaFailoverThread can take too long to stop and leak out of unit tests. Key: SOLR-9846 URL: https://issues.apache.org/jira/browse/SOLR-9846 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller Assignee: Mark Miller We should interrupt it on close. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9845) OverseerAutoReplicaFailoverThread can take too long to stop and leak out of unit tests.
Mark Miller created SOLR-9845: - Summary: OverseerAutoReplicaFailoverThread can take too long to stop and leak out of unit tests. Key: SOLR-9845 URL: https://issues.apache.org/jira/browse/SOLR-9845 Project: Solr Issue Type: Test Security Level: Public (Default Security Level. Issues are Public) Reporter: Mark Miller Assignee: Mark Miller We should interrupt it on close. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 547 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/547/ Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.cloud.hdfs.HdfsRecoveryZkTest Error Message: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43) at org.apache.solr.update.HdfsTransactionLog.(HdfsTransactionLog.java:130) at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202) at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:137) at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:94) at org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandler2.java:102) at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:705) at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:767) at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1006) at org.apache.solr.core.SolrCore.(SolrCore.java:871) at org.apache.solr.core.SolrCore.(SolrCore.java:775) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:842) at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Stack Trace: java.lang.AssertionError: ObjectTracker found 1 object(s) that were not released!!! [HdfsTransactionLog] org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException at org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43) at org.apache.solr.update.HdfsTransactionLog.(HdfsTransactionLog.java:130) at org.apache.solr.update.HdfsUpdateLog.init(HdfsUpdateLog.java:202) at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:137) at org.apache.solr.update.UpdateHandler.(UpdateHandler.java:94) at org.apache.solr.update.DirectUpdateHandler2.(DirectUpdateHandler2.java:102) at sun.reflect.GeneratedConstructorAccessor192.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.solr.core.SolrCore.createInstance(SolrCore.java:705) at org.apache.solr.core.SolrCore.createUpdateHandler(SolrCore.java:767) at org.apache.solr.core.SolrCore.initUpdateHandler(SolrCore.java:1006) at org.apache.solr.core.SolrCore.(SolrCore.java:871) at org.apache.solr.core.SolrCore.(SolrCore.java:775) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:842) at org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) at __randomizedtesting.SeedInfo.seed([D9C191A156B42503]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNull(Assert.java:551) at org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:266) at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:870) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at
[jira] [Updated] (SOLR-9844) Display fieldCache total size information
[ https://issues.apache.org/jira/browse/SOLR-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-9844: Attachment: Screen Shot 2016-12-11 at 6.00.36 PM.png SOLR-9844.patch Patch which addresses both the issues mentioned. Attaching a screenshot of how it will look after the patch is committed. The encoding issue reported on SOLR-9297 still exists. > Display fieldCache total size information > - > > Key: SOLR-9844 > URL: https://issues.apache.org/jira/browse/SOLR-9844 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Assignee: Varun Thacker >Priority: Minor > Attachments: SOLR-9844.patch, Screen Shot 2016-12-11 at 6.00.36 PM.png > > > We know the size of each entry in the fieldCache and display that on the > admin UI screen under Plugin/Stats -> fieldCache currently. > We also show the total entries count. Lets also show the total size of the > cache . > This is valuable information to tell users how much heap memory is being > occupied by the fieldCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down
[ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller reassigned SOLR-1953: - Assignee: Mark Miller > It may be possible for temporary files to accumulator until the Solr process > is shut down > - > > Key: SOLR-1953 > URL: https://issues.apache.org/jira/browse/SOLR-1953 > Project: Solr > Issue Type: Bug > Components: update >Affects Versions: 1.4.1, 1.5 >Reporter: Karl Wright >Assignee: Mark Miller > Labels: file_leak, memory_leak > Attachments: SOLR-1953.patch, SOLR-1953.patch > > > While researching SOLR-1951, the behavior of commons-fileupload in handling > multipart form posts came into question. commons-fileupload creates a > temporary file for the main content area of such posts, and purportedly has a > background thread which cleans up these files. However, Mark Miller > discovered that the javadoc in this matter may be incorrect, and that > commons-fileupload may in fact just be adding files to the JVM's list of > files needing cleanup on exit. > If so, this will show up in two ways: first, temporary files will accumulate > in the java.io.tmpdir area. Second, non-heap memory for the JVM will slowly > increase over time (since the file pointers the JVM tracks in this way are > not kept in the java heap). > I will attach a potential fix; however, this ticket should be viewed as a > workitem for the need for further research in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down
[ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740677#comment-15740677 ] Mark Miller commented on SOLR-1953: --- Wow this is old. Lost it I guess. > It may be possible for temporary files to accumulator until the Solr process > is shut down > - > > Key: SOLR-1953 > URL: https://issues.apache.org/jira/browse/SOLR-1953 > Project: Solr > Issue Type: Bug > Components: update >Affects Versions: 1.4.1, 1.5 >Reporter: Karl Wright > Labels: file_leak, memory_leak > Attachments: SOLR-1953.patch, SOLR-1953.patch > > > While researching SOLR-1951, the behavior of commons-fileupload in handling > multipart form posts came into question. commons-fileupload creates a > temporary file for the main content area of such posts, and purportedly has a > background thread which cleans up these files. However, Mark Miller > discovered that the javadoc in this matter may be incorrect, and that > commons-fileupload may in fact just be adding files to the JVM's list of > files needing cleanup on exit. > If so, this will show up in two ways: first, temporary files will accumulate > in the java.io.tmpdir area. Second, non-heap memory for the JVM will slowly > increase over time (since the file pointers the JVM tracks in this way are > not kept in the java heap). > I will attach a potential fix; however, this ticket should be viewed as a > workitem for the need for further research in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-8733) versions=true on Optimistic Concurrency updates doesn't work with in multi shard collections
[
https://issues.apache.org/jira/browse/SOLR-8733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740675#comment-15740675
]
Jun Wu commented on SOLR-8733:
--
NO one cares this issue ?
> versions=true on Optimistic Concurrency updates doesn't work with in multi
> shard collections
>
>
> Key: SOLR-8733
> URL: https://issues.apache.org/jira/browse/SOLR-8733
> Project: Solr
> Issue Type: Bug
>Reporter: Hoss Man
> Attachments: SOLR-8733_incomplete_test.patch,
> TestReturnedVersions.java
>
>
> Noticed this while working on SOLR-445...
> the {{versions=true}} param is documented as an option that can be used when
> doing updates if you care about opportunistic concurrency so that you know
> for certain which {{_version_}} was associated with each update you send, w/o
> needing to issue a subsequent {{/get}} request for the updated documents
> (which might return some _newer_ version if another client did an update
> immediately after you...
> https://cwiki.apache.org/confluence/display/solr/Updating+Parts+of+Documents
> Ironically, even though DistributedUpdateProcessor is where all of the
> opportunistic concurrency code lives, and the logic for {{versions=true}} is
> handled, it does nothing to ensure that the results are populated when
> requests are forwarded to other leaders - instead, the code seems to just
> assume all updates have their version assigned locally, so they wind up being
> "0" ...
> Single node example of it working properly...
> {noformat}
> $ bin/solr -e techproducts -noprompt
> ...
> $ curl -H 'Content-Type: application/json'
> 'http://localhost:8983/solr/techproducts/update?versions=true' --data-binary
> '[{"id":"abc!111","foo_s":"one"}, {"id":"XYZ!222","foo_s":"two"}]'
> {"responseHeader":{"status":0,"QTime":10},"adds":["abc!111",1527095970400043008,"XYZ!222",1527095970402140160]}
> {noformat}
> Multinode multi-shard example of current broken behavior...
> {noformat}
> $ bin/solr -e cloud -noprompt
> ...
> $ curl -H 'Content-Type: application/json'
> 'http://localhost:8983/solr/gettingstarted/update?versions=true'
> --data-binary '[{"id":"abc!111","foo_s":"one"},
> {"id":"XYZ!222","foo_s":"two"}]'
> {"responseHeader":{"status":0,"QTime":32},"adds":["abc!111",0,"XYZ!222",0]}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9824) Documents indexed in bulk are replicated using too many HTTP requests
[
https://issues.apache.org/jira/browse/SOLR-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller updated SOLR-9824:
--
Attachment: SOLR-9824.patch
More polish and lots more testing done.
I did end up removing the polling setting on an updaterequest since it no
longer takes effect.
Still testing, but this is banging into shape.
> Documents indexed in bulk are replicated using too many HTTP requests
> -
>
> Key: SOLR-9824
> URL: https://issues.apache.org/jira/browse/SOLR-9824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 6.3
>Reporter: David Smiley
> Attachments: SOLR-9824.patch, SOLR-9824.patch, SOLR-9824.patch,
> SOLR-9824.patch, SOLR-9824.patch, SOLR-9824.patch
>
>
> This takes awhile to explain; bear with me. While working on bulk indexing
> small documents, I looked at the logs of my SolrCloud nodes. I noticed that
> shards would see an /update log message every ~6ms which is *way* too much.
> These are requests from one shard (that isn't a leader/replica for these docs
> but the recipient from my client) to the target shard leader (no additional
> replicas). One might ask why I'm not sending docs to the right shard in the
> first place; I have a reason but it's besides the point -- there's a real
> Solr perf problem here and this probably applies equally to
> replicationFactor>1 situations too. I could turn off the logs but that would
> hide useful stuff, and it's disconcerting to me that so many short-lived HTTP
> requests are happening, somehow at the bequest of DistributedUpdateProcessor.
> After lots of analysis and debugging and hair pulling, I finally figured it
> out.
> In SOLR-7333 ([~tpot]) introduced an optimization called
> {{UpdateRequest.isLastDocInBatch()}} in which ConcurrentUpdateSolrClient will
> poll with a '0' timeout to the internal queue, so that it can close the
> connection without it hanging around any longer than needed. This part makes
> sense to me. Currently the only spot that has the smarts to set this flag is
> {{JavaBinUpdateRequestCodec.unmarshal.readOuterMostDocIterator()}} at the
> last document. So if a shard received docs in a javabin stream (but not
> other formats) one would expect the _last_ document to have this flag.
> There's even a test. Docs without this flag get the default poll time; for
> javabin it's 25ms. Okay.
> I _suspect_ that if someone used CloudSolrClient or HttpSolrClient to send
> javabin data in a batch, the intended efficiencies of SOLR-7333 would apply.
> I didn't try. In my case, I'm using ConcurrentUpdateSolrClient (and BTW
> DistributedUpdateProcessor uses CUSC too). CUSC uses the RequestWriter
> (defaulting to javabin) to send each document separately without any leading
> marker or trailing marker. For the XML format by comparison, there is a
> leading and trailing marker ( ... ). Since there's no outer
> container for the javabin unmarshalling to detect the last document, it marks
> _every_ document as {{req.lastDocInBatch()}}! Ouch!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-6.x - Build # 596 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/596/ 1 tests failed. FAILED: org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test Error Message: Stack Trace: java.util.concurrent.TimeoutException at __randomizedtesting.SeedInfo.seed([B5BBBF67A6FE1BC5:3DEF80BD0802763D]:0) at org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1251) at org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:697) at org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica.test(TestLeaderElectionWithEmptyReplica.java:97) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 12270 lines...] [junit4] Suite: org.apache.solr.cloud.TestLeaderElectionWithEmptyReplica [junit4] 2> Creating dataDir:
[jira] [Commented] (SOLR-9844) Display fieldCache total size information
[
https://issues.apache.org/jira/browse/SOLR-9844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740548#comment-15740548
]
Varun Thacker commented on SOLR-9844:
-
Also its annoying that the current entries show up like this
{code}
entry#0:
'org.apache.lucene.index.SegmentCoreReaders@20836290'=>'cat',class
org.apache.lucene.index.SortedDocValues,0.5=>org.apache.solr.uninverting.FieldCacheImpl$SortedDocValuesImpl#1504618891
(size =~ 288 bytes)
{code}
I propose to simplify it to
{code}
entry#0 field='cat' segment=_0 size='~288 bytes'
{code}
So this will tell us the field , segment and size of the entry.
> Display fieldCache total size information
> -
>
> Key: SOLR-9844
> URL: https://issues.apache.org/jira/browse/SOLR-9844
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Minor
>
> We know the size of each entry in the fieldCache and display that on the
> admin UI screen under Plugin/Stats -> fieldCache currently.
> We also show the total entries count. Lets also show the total size of the
> cache .
> This is valuable information to tell users how much heap memory is being
> occupied by the fieldCache.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-9844) Display fieldCache total size information
Varun Thacker created SOLR-9844: --- Summary: Display fieldCache total size information Key: SOLR-9844 URL: https://issues.apache.org/jira/browse/SOLR-9844 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker Assignee: Varun Thacker Priority: Minor We know the size of each entry in the fieldCache and display that on the admin UI screen under Plugin/Stats -> fieldCache currently. We also show the total entries count. Lets also show the total size of the cache . This is valuable information to tell users how much heap memory is being occupied by the fieldCache. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+147) - Build # 18505 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18505/ Java: 32bit/jdk-9-ea+147 -server -XX:+UseSerialGC 1 tests failed. FAILED: org.apache.solr.handler.component.SpellCheckComponentTest.testNumericQuery Error Message: List size mismatch @ spellcheck/suggestions Stack Trace: java.lang.RuntimeException: List size mismatch @ spellcheck/suggestions at __randomizedtesting.SeedInfo.seed([808F37509846446F:8BA36090078FF8C0]:0) at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:906) at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:853) at org.apache.solr.handler.component.SpellCheckComponentTest.testNumericQuery(SpellCheckComponentTest.java:154) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:538) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.base/java.lang.Thread.run(Thread.java:844) Build Log: [...truncated 11644 lines...] [junit4] Suite:
[jira] [Commented] (LUCENE-7580) Spans tree scoring
[ https://issues.apache.org/jira/browse/LUCENE-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740406#comment-15740406 ] Paul Elschot commented on LUCENE-7580: -- Some scientific articles on this subject: Metzler, Donald, and W. Bruce Croft. "A Markov random field model for term dependencies." Proceedings of the 28th annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 2005. In section 2.3 they use terms and ordered and unordered phrases The ranking function is a weighted linear combination for these. The optimal weights are about 80/10/10 for simple terms, unordered, and ordered. Here this led to the use of a weighting factor non matching occurrences. They also found that the minimum distance is the best indicator of relevance. Bendersky, Michael, and W. Bruce Croft. "Modeling Higher-Order Term Dependencies in Information Retrieval using Query Hypergraphs" SIGIR'12. The concepts there can be nested, like span queries. The approach there is much more general. For example: - Table 2 shows the use of the frequency of a concept in various collections to determine its weight. - In section 2.4.2 there is an indication that the slop factor needs attention: "... the existing term proximity measures usually capture close, sentence-level, co-occurrences of the query terms ... The dependency range is much longer for concept dependencies." Blanco, Roi, and Paolo Boldi. "Extending BM25 with multiple query operators." Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval. ACM, 2012. This scores regions with BM25F. > Spans tree scoring > -- > > Key: LUCENE-7580 > URL: https://issues.apache.org/jira/browse/LUCENE-7580 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: master (7.0) >Reporter: Paul Elschot >Priority: Minor > Fix For: 6.x > > Attachments: LUCENE-7580.patch, LUCENE-7580.patch > > > Recurse the spans tree to compose a score based on the type of subqueries and > what matched -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-7580) Spans tree scoring
[ https://issues.apache.org/jira/browse/LUCENE-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740402#comment-15740402 ] Paul Elschot commented on LUCENE-7580: -- This adds a nonMatchSlop attribute to SpanNearQuery, and drops the nonMatchSlopFactor argument from SpansTreeQuery. nonMatchSlop is the distance for determining a slop factor that is to be used for non matching occurrences of a SpanNearQuery. Smaller values for this distance will increase the score contribution of non matching occurrences via SimScorer.computeSlopFactor() But smaller values for this distance, i.e. higher score contribution of non matching occurrences, may lead to a scoring inconsistency between two span near queries that only differ in the allowed slop. For example consider query A with a smaller allowed slop and query B with a larger one. For query B there can be more matches, and these should increase the score of B when compared to the score of A. So for each extra match at B, the non matching score for query A should be lower than the matching score for query B. This may not be the case when the non matching score contribution is too high. To have consistent scoring between two such queries, choose a non matching slop that is larger than the largest allowed match slop, and provide that non matching slop to both queries. In case this consistency is not needed, nonMatchSlop can be chosen to be somewhat larger than the maximum allowed match slop. This nonMatchSlop is used in SpansTreeWeight to compute a minimal nested slop factor from the maximum possible slops that can occur in a SpanQuery for the nested SpanNearQueries and for nested SpanOrQueries with distance. Finally, this minimal nested slop factor is used as the weight for scoring non matching terms. The default nonMatchSlop for SpanNearQuery is large, Integer.MAX_VALUE/2. Therefore by default non matching occurrences have no real score contribution. > Spans tree scoring > -- > > Key: LUCENE-7580 > URL: https://issues.apache.org/jira/browse/LUCENE-7580 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: master (7.0) >Reporter: Paul Elschot >Priority: Minor > Fix For: 6.x > > Attachments: LUCENE-7580.patch, LUCENE-7580.patch > > > Recurse the spans tree to compose a score based on the type of subqueries and > what matched -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-7580) Spans tree scoring
[ https://issues.apache.org/jira/browse/LUCENE-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740402#comment-15740402 ] Paul Elschot edited comment on LUCENE-7580 at 12/11/16 9:44 PM: Compared to the previous patch, this adds a nonMatchSlop attribute to SpanNearQuery, and drops the nonMatchSlopFactor argument from SpansTreeQuery. nonMatchSlop is the distance for determining a slop factor that is to be used for non matching occurrences of a SpanNearQuery. Smaller values for this distance will increase the score contribution of non matching occurrences via SimScorer.computeSlopFactor() But smaller values for this distance, i.e. higher score contribution of non matching occurrences, may lead to a scoring inconsistency between two span near queries that only differ in the allowed slop. For example consider query A with a smaller allowed slop and query B with a larger one. For query B there can be more matches, and these should increase the score of B when compared to the score of A. So for each extra match at B, the non matching score for query A should be lower than the matching score for query B. This may not be the case when the non matching score contribution is too high. To have consistent scoring between two such queries, choose a non matching slop that is larger than the largest allowed match slop, and provide that non matching slop to both queries. In case this consistency is not needed, nonMatchSlop can be chosen to be somewhat larger than the maximum allowed match slop. This nonMatchSlop is used in SpansTreeWeight to compute a minimal nested slop factor from the maximum possible slops that can occur in a SpanQuery for the nested SpanNearQueries and for nested SpanOrQueries with distance. Finally, this minimal nested slop factor is used as the weight for scoring non matching terms. The default nonMatchSlop for SpanNearQuery is large, Integer.MAX_VALUE/2. Therefore by default non matching occurrences have no real score contribution. was (Author: paul.elsc...@xs4all.nl): This adds a nonMatchSlop attribute to SpanNearQuery, and drops the nonMatchSlopFactor argument from SpansTreeQuery. nonMatchSlop is the distance for determining a slop factor that is to be used for non matching occurrences of a SpanNearQuery. Smaller values for this distance will increase the score contribution of non matching occurrences via SimScorer.computeSlopFactor() But smaller values for this distance, i.e. higher score contribution of non matching occurrences, may lead to a scoring inconsistency between two span near queries that only differ in the allowed slop. For example consider query A with a smaller allowed slop and query B with a larger one. For query B there can be more matches, and these should increase the score of B when compared to the score of A. So for each extra match at B, the non matching score for query A should be lower than the matching score for query B. This may not be the case when the non matching score contribution is too high. To have consistent scoring between two such queries, choose a non matching slop that is larger than the largest allowed match slop, and provide that non matching slop to both queries. In case this consistency is not needed, nonMatchSlop can be chosen to be somewhat larger than the maximum allowed match slop. This nonMatchSlop is used in SpansTreeWeight to compute a minimal nested slop factor from the maximum possible slops that can occur in a SpanQuery for the nested SpanNearQueries and for nested SpanOrQueries with distance. Finally, this minimal nested slop factor is used as the weight for scoring non matching terms. The default nonMatchSlop for SpanNearQuery is large, Integer.MAX_VALUE/2. Therefore by default non matching occurrences have no real score contribution. > Spans tree scoring > -- > > Key: LUCENE-7580 > URL: https://issues.apache.org/jira/browse/LUCENE-7580 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: master (7.0) >Reporter: Paul Elschot >Priority: Minor > Fix For: 6.x > > Attachments: LUCENE-7580.patch, LUCENE-7580.patch > > > Recurse the spans tree to compose a score based on the type of subqueries and > what matched -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7580) Spans tree scoring
[ https://issues.apache.org/jira/browse/LUCENE-7580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Elschot updated LUCENE-7580: - Attachment: LUCENE-7580.patch Patch of 11 Dec 2016. Add automatically determining a weight for non matching terms. > Spans tree scoring > -- > > Key: LUCENE-7580 > URL: https://issues.apache.org/jira/browse/LUCENE-7580 > Project: Lucene - Core > Issue Type: Improvement > Components: core/search >Affects Versions: master (7.0) >Reporter: Paul Elschot >Priority: Minor > Fix For: 6.x > > Attachments: LUCENE-7580.patch, LUCENE-7580.patch > > > Recurse the spans tree to compose a score based on the type of subqueries and > what matched -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9824) Documents indexed in bulk are replicated using too many HTTP requests
[
https://issues.apache.org/jira/browse/SOLR-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller updated SOLR-9824:
--
Attachment: SOLR-9824.patch
This approach is testing out really well. I'll start cleaning up the patch.
Still a bit more testing I'd like to do as well.
> Documents indexed in bulk are replicated using too many HTTP requests
> -
>
> Key: SOLR-9824
> URL: https://issues.apache.org/jira/browse/SOLR-9824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 6.3
>Reporter: David Smiley
> Attachments: SOLR-9824.patch, SOLR-9824.patch, SOLR-9824.patch,
> SOLR-9824.patch, SOLR-9824.patch
>
>
> This takes awhile to explain; bear with me. While working on bulk indexing
> small documents, I looked at the logs of my SolrCloud nodes. I noticed that
> shards would see an /update log message every ~6ms which is *way* too much.
> These are requests from one shard (that isn't a leader/replica for these docs
> but the recipient from my client) to the target shard leader (no additional
> replicas). One might ask why I'm not sending docs to the right shard in the
> first place; I have a reason but it's besides the point -- there's a real
> Solr perf problem here and this probably applies equally to
> replicationFactor>1 situations too. I could turn off the logs but that would
> hide useful stuff, and it's disconcerting to me that so many short-lived HTTP
> requests are happening, somehow at the bequest of DistributedUpdateProcessor.
> After lots of analysis and debugging and hair pulling, I finally figured it
> out.
> In SOLR-7333 ([~tpot]) introduced an optimization called
> {{UpdateRequest.isLastDocInBatch()}} in which ConcurrentUpdateSolrClient will
> poll with a '0' timeout to the internal queue, so that it can close the
> connection without it hanging around any longer than needed. This part makes
> sense to me. Currently the only spot that has the smarts to set this flag is
> {{JavaBinUpdateRequestCodec.unmarshal.readOuterMostDocIterator()}} at the
> last document. So if a shard received docs in a javabin stream (but not
> other formats) one would expect the _last_ document to have this flag.
> There's even a test. Docs without this flag get the default poll time; for
> javabin it's 25ms. Okay.
> I _suspect_ that if someone used CloudSolrClient or HttpSolrClient to send
> javabin data in a batch, the intended efficiencies of SOLR-7333 would apply.
> I didn't try. In my case, I'm using ConcurrentUpdateSolrClient (and BTW
> DistributedUpdateProcessor uses CUSC too). CUSC uses the RequestWriter
> (defaulting to javabin) to send each document separately without any leading
> marker or trailing marker. For the XML format by comparison, there is a
> leading and trailing marker ( ... ). Since there's no outer
> container for the javabin unmarshalling to detect the last document, it marks
> _every_ document as {{req.lastDocInBatch()}}! Ouch!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-6.x-Windows (32bit/jdk1.8.0_112) - Build # 616 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/616/ Java: 32bit/jdk1.8.0_112 -server -XX:+UseG1GC 1 tests failed. FAILED: junit.framework.TestSuite.org.apache.solr.handler.dataimport.TestEphemeralCache Error Message: Could not remove the following files (in the order of attempts): C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001\collection1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001\collection1 C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001 Stack Trace: java.io.IOException: Could not remove the following files (in the order of attempts): C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001\collection1: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001\collection1 C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001: java.nio.file.DirectoryNotEmptyException: C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\core-home-001 at __randomizedtesting.SeedInfo.seed([DA8B79FCBB35CCD6]:0) at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323) at org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216) at com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 17514 lines...] [junit4] Suite: org.apache.solr.handler.dataimport.TestEphemeralCache [junit4] 2> Creating dataDir: C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J0\temp\solr.handler.dataimport.TestEphemeralCache_DA8B79FCBB35CCD6-001\init-core-data-001 [junit4] 2> 33167 T299 oas.SolrTestCaseJ4.buildSSLConfig Randomized ssl (false) and clientAuth (false) via: @org.apache.solr.util.RandomizeSSL(reason=, value=NaN, ssl=NaN, clientAuth=NaN) [junit4] 2> 33229 T299 oas.SolrTestCaseJ4.initCore initCore [junit4] 2> 33259 T299 oasc.SolrConfig. Using Lucene MatchVersion: 6.4.0 [junit4] 2> 33274 T299 oass.IndexSchema.readSchema [null] Schema name=dih_test [junit4] 2> 33291 T299 oass.IndexSchema.readSchema WARN [null] default search field in schema is desc. WARNING: Deprecated, please use 'df' on request instead. [junit4] 2> 33292 T299 oass.IndexSchema.readSchema WARN [null] query parser default operator is OR. WARNING: Deprecated, please use 'q.op' on request instead. [junit4] 2> 33292 T299 oass.IndexSchema.readSchema Loaded schema dih_test/4.0 with uniqueid field id [junit4] 2> 33300 T299 oasu.UpdateShardHandler. Creating UpdateShardHandler HTTP client with params: socketTimeout=3=3=true [junit4] 2> 33322 T302 oasc.SolrConfig. Using Lucene MatchVersion: 6.4.0 [junit4] 2> 6 T302 oass.IndexSchema.readSchema [collection1] Schema name=dih_test [junit4] 2> 33344 T302 oass.IndexSchema.readSchema WARN [collection1] default search field
[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 225 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/225/ 4 tests failed. FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchAddsWithDelete Error Message: Timeout while trying to assert number of documents @ target_collection Stack Trace: java.lang.AssertionError: Timeout while trying to assert number of documents @ target_collection at __randomizedtesting.SeedInfo.seed([235D4743C0934EBA:54473EE7688A4F96]:0) at org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271) at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchAddsWithDelete(CdcrReplicationDistributedZkTest.java:532) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Updated] (SOLR-9824) Documents indexed in bulk are replicated using too many HTTP requests
[
https://issues.apache.org/jira/browse/SOLR-9824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller updated SOLR-9824:
--
Attachment: SOLR-9824.patch
Here is where I am currently at. At the moment it builds on SOLR-9842.patch,
which allows to be sure we shutdown the CUSS clients in all cases.
Needs more testing, but this path may work.
> Documents indexed in bulk are replicated using too many HTTP requests
> -
>
> Key: SOLR-9824
> URL: https://issues.apache.org/jira/browse/SOLR-9824
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: 6.3
>Reporter: David Smiley
> Attachments: SOLR-9824.patch, SOLR-9824.patch, SOLR-9824.patch,
> SOLR-9824.patch
>
>
> This takes awhile to explain; bear with me. While working on bulk indexing
> small documents, I looked at the logs of my SolrCloud nodes. I noticed that
> shards would see an /update log message every ~6ms which is *way* too much.
> These are requests from one shard (that isn't a leader/replica for these docs
> but the recipient from my client) to the target shard leader (no additional
> replicas). One might ask why I'm not sending docs to the right shard in the
> first place; I have a reason but it's besides the point -- there's a real
> Solr perf problem here and this probably applies equally to
> replicationFactor>1 situations too. I could turn off the logs but that would
> hide useful stuff, and it's disconcerting to me that so many short-lived HTTP
> requests are happening, somehow at the bequest of DistributedUpdateProcessor.
> After lots of analysis and debugging and hair pulling, I finally figured it
> out.
> In SOLR-7333 ([~tpot]) introduced an optimization called
> {{UpdateRequest.isLastDocInBatch()}} in which ConcurrentUpdateSolrClient will
> poll with a '0' timeout to the internal queue, so that it can close the
> connection without it hanging around any longer than needed. This part makes
> sense to me. Currently the only spot that has the smarts to set this flag is
> {{JavaBinUpdateRequestCodec.unmarshal.readOuterMostDocIterator()}} at the
> last document. So if a shard received docs in a javabin stream (but not
> other formats) one would expect the _last_ document to have this flag.
> There's even a test. Docs without this flag get the default poll time; for
> javabin it's 25ms. Okay.
> I _suspect_ that if someone used CloudSolrClient or HttpSolrClient to send
> javabin data in a batch, the intended efficiencies of SOLR-7333 would apply.
> I didn't try. In my case, I'm using ConcurrentUpdateSolrClient (and BTW
> DistributedUpdateProcessor uses CUSC too). CUSC uses the RequestWriter
> (defaulting to javabin) to send each document separately without any leading
> marker or trailing marker. For the XML format by comparison, there is a
> leading and trailing marker ( ... ). Since there's no outer
> container for the javabin unmarshalling to detect the last document, it marks
> _every_ document as {{req.lastDocInBatch()}}! Ouch!
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-9842) UpdateRequestProcessors have no way to guarantee the closing of resources used for a request.
[ https://issues.apache.org/jira/browse/SOLR-9842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Miller updated SOLR-9842: -- Attachment: SOLR-9842.patch > UpdateRequestProcessors have no way to guarantee the closing of resources > used for a request. > - > > Key: SOLR-9842 > URL: https://issues.apache.org/jira/browse/SOLR-9842 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller > Attachments: SOLR-9842.patch > > > You cannot count on finish being called for an UpdateRequestProcessor because > whether or not it's called in the face of errors is dependent on the other > processors in the chain. > We should make UpdateRequestProcessor closeable and implement this so that > close is guaranteed to be called. I'm playing around with this in a patch for > another issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+147) - Build # 18502 - Unstable!
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18502/ Java: 32bit/jdk-9-ea+147 -client -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.handler.component.SpellCheckComponentTest.test Error Message: List size mismatch @ spellcheck/suggestions Stack Trace: java.lang.RuntimeException: List size mismatch @ spellcheck/suggestions at __randomizedtesting.SeedInfo.seed([C8194970D1517B9:84D5AB4DA3E97A41]:0) at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:906) at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:853) at org.apache.solr.handler.component.SpellCheckComponentTest.test(SpellCheckComponentTest.java:147) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:538) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at java.base/java.lang.Thread.run(Thread.java:844) Build Log: [...truncated 11250 lines...] [junit4] Suite: org.apache.solr.handler.component.SpellCheckComponentTest [junit4] 2> Creating
[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1179 - Still Unstable
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1179/ 6 tests failed. FAILED: org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchBoundaries Error Message: Timeout waiting for CDCR replication to complete @source_collection:shard1 Stack Trace: java.lang.RuntimeException: Timeout waiting for CDCR replication to complete @source_collection:shard1 at __randomizedtesting.SeedInfo.seed([47807533F98A83F4:65AFAC66802E4213]:0) at org.apache.solr.cloud.BaseCdcrDistributedZkTest.waitForReplicationToComplete(BaseCdcrDistributedZkTest.java:795) at org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testBatchBoundaries(CdcrReplicationDistributedZkTest.java:557) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907) at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943) at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Comment Edited] (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down
[ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739469#comment-15739469 ] Gilad Moscovitch edited comment on SOLR-1953 at 12/11/16 11:21 AM: --- Hey. Do you think it is somehow related to the next issue (using solr 6.2.1): An increasing 'slab' memory consumption. When running `slabtop` we get a 14g under 'dentry' and counting. was (Author: moscovig): Hey. Do you think it is somehow related to the next issue: An increasing 'slab' memory consumption. When running `slabtop` we get a 14g under 'dentry' and counting. > It may be possible for temporary files to accumulator until the Solr process > is shut down > - > > Key: SOLR-1953 > URL: https://issues.apache.org/jira/browse/SOLR-1953 > Project: Solr > Issue Type: Bug > Components: update >Affects Versions: 1.4.1, 1.5 >Reporter: Karl Wright > Labels: file_leak, memory_leak > Attachments: SOLR-1953.patch, SOLR-1953.patch > > > While researching SOLR-1951, the behavior of commons-fileupload in handling > multipart form posts came into question. commons-fileupload creates a > temporary file for the main content area of such posts, and purportedly has a > background thread which cleans up these files. However, Mark Miller > discovered that the javadoc in this matter may be incorrect, and that > commons-fileupload may in fact just be adding files to the JVM's list of > files needing cleanup on exit. > If so, this will show up in two ways: first, temporary files will accumulate > in the java.io.tmpdir area. Second, non-heap memory for the JVM will slowly > increase over time (since the file pointers the JVM tracks in this way are > not kept in the java heap). > I will attach a potential fix; however, this ticket should be viewed as a > workitem for the need for further research in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-7590) Add DocValues statistics helpers
[
https://issues.apache.org/jira/browse/LUCENE-7590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shai Erera updated LUCENE-7590:
---
Attachment: LUCENE-7590.patch
First patch adds numeric statistics. I'd appreciate comments about it before I
add support for sorted-numeric (including, whether we should!).
Note that I chose to take either a field or {{ValueSource}}. The latter gives
some flexibility by allowing users to pass an arbitrary VS over e.g. an
{{Expression}} over a numeric DV field.
This, as far as I could tell, does not apply to {{SortedNumericDV}}, or at
least I couldn't find an existing {{ValueSource}} implementation (like
{{LongFieldSource}}) for {{SortedNumericDV}}.
If this approach looks good, I'd like to refactor the class so that it's easy
to share/reuse code between Long and Double NDV fields.
> Add DocValues statistics helpers
>
>
> Key: LUCENE-7590
> URL: https://issues.apache.org/jira/browse/LUCENE-7590
> Project: Lucene - Core
> Issue Type: New Feature
> Components: modules/misc
>Reporter: Shai Erera
>Assignee: Shai Erera
> Attachments: LUCENE-7590.patch
>
>
> I think it can be useful to have DocValues statistics helpers, that can allow
> users to query for the min/max/avg etc. stats of a DV field. In this issue
> I'd like to cover numeric DV, but there's no reason not to add it to other DV
> types too.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-7590) Add DocValues statistics helpers
Shai Erera created LUCENE-7590: -- Summary: Add DocValues statistics helpers Key: LUCENE-7590 URL: https://issues.apache.org/jira/browse/LUCENE-7590 Project: Lucene - Core Issue Type: New Feature Components: modules/misc Reporter: Shai Erera Assignee: Shai Erera I think it can be useful to have DocValues statistics helpers, that can allow users to query for the min/max/avg etc. stats of a DV field. In this issue I'd like to cover numeric DV, but there's no reason not to add it to other DV types too. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-1953) It may be possible for temporary files to accumulator until the Solr process is shut down
[ https://issues.apache.org/jira/browse/SOLR-1953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739469#comment-15739469 ] Gilad Moscovitch commented on SOLR-1953: Hey. Do you think it is somehow related to the next issue: An increasing 'slab' memory consumption. When running `slabtop` we get a 14g under 'dentry' and counting. > It may be possible for temporary files to accumulator until the Solr process > is shut down > - > > Key: SOLR-1953 > URL: https://issues.apache.org/jira/browse/SOLR-1953 > Project: Solr > Issue Type: Bug > Components: update >Affects Versions: 1.4.1, 1.5 >Reporter: Karl Wright > Labels: file_leak, memory_leak > Attachments: SOLR-1953.patch, SOLR-1953.patch > > > While researching SOLR-1951, the behavior of commons-fileupload in handling > multipart form posts came into question. commons-fileupload creates a > temporary file for the main content area of such posts, and purportedly has a > background thread which cleans up these files. However, Mark Miller > discovered that the javadoc in this matter may be incorrect, and that > commons-fileupload may in fact just be adding files to the JVM's list of > files needing cleanup on exit. > If so, this will show up in two ways: first, temporary files will accumulate > in the java.io.tmpdir area. Second, non-heap memory for the JVM will slowly > increase over time (since the file pointers the JVM tracks in this way are > not kept in the java heap). > I will attach a potential fix; however, this ticket should be viewed as a > workitem for the need for further research in this area. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5882) Introduce score local parameter for {!parent} query parser
[
https://issues.apache.org/jira/browse/SOLR-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739416#comment-15739416
]
Vladimir Strugatsky commented on SOLR-5882:
---
Indeed, I've tried
{code}
"q": "{!parent which=isParent:true score=max} +_query_:\"{!edismax
qf=text}copy\"",
"boost": "if(termfreq(statusFacet,\"Supporting\"),0.3,1.0)",
{code}
with success, but this breaks highlighting per
https://issues.apache.org/jira/browse/SOLR-2632. It appears that the
combination of _query_, boost and highlighting is problematic.
> Introduce score local parameter for {!parent} query parser
> --
>
> Key: SOLR-5882
> URL: https://issues.apache.org/jira/browse/SOLR-5882
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 4.8
>Reporter: Andrey Kudryavtsev
>Assignee: Mikhail Khludnev
> Fix For: 5.3
>
> Attachments: SOLR-5882.patch, SOLR-5882.patch
>
>
> I propose to have ability to configure scoring mode by *optional* local
> parameter
> Syntax for parent queries is:
> {code:borderStyle=solid}
> {!parent which=type:parent score=none|avg|max|total|min}...
> Capital case for score values is also accepted
> {code}
> Child query enables scoring by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5882) Introduce score local parameter for {!parent} query parser
[
https://issues.apache.org/jira/browse/SOLR-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739380#comment-15739380
]
Vladimir Strugatsky commented on SOLR-5882:
---
I have a missing parenthesis at the end of the boost - this is a typo, but the
boost has no effect with or without the typo.
> Introduce score local parameter for {!parent} query parser
> --
>
> Key: SOLR-5882
> URL: https://issues.apache.org/jira/browse/SOLR-5882
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 4.8
>Reporter: Andrey Kudryavtsev
>Assignee: Mikhail Khludnev
> Fix For: 5.3
>
> Attachments: SOLR-5882.patch, SOLR-5882.patch
>
>
> I propose to have ability to configure scoring mode by *optional* local
> parameter
> Syntax for parent queries is:
> {code:borderStyle=solid}
> {!parent which=type:parent score=none|avg|max|total|min}...
> Capital case for score values is also accepted
> {code}
> Child query enables scoring by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-5882) Introduce score local parameter for {!parent} query parser
[
https://issues.apache.org/jira/browse/SOLR-5882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15739350#comment-15739350
]
Mikhail Khludnev commented on SOLR-5882:
{{boost}} parameter is e?dismax's feature. See
https://cwiki.apache.org/confluence/display/solr/The+Extended+DisMax+Query+Parser
> Introduce score local parameter for {!parent} query parser
> --
>
> Key: SOLR-5882
> URL: https://issues.apache.org/jira/browse/SOLR-5882
> Project: Solr
> Issue Type: New Feature
>Affects Versions: 4.8
>Reporter: Andrey Kudryavtsev
>Assignee: Mikhail Khludnev
> Fix For: 5.3
>
> Attachments: SOLR-5882.patch, SOLR-5882.patch
>
>
> I propose to have ability to configure scoring mode by *optional* local
> parameter
> Syntax for parent queries is:
> {code:borderStyle=solid}
> {!parent which=type:parent score=none|avg|max|total|min}...
> Capital case for score values is also accepted
> {code}
> Child query enables scoring by default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
