[JENKINS] Lucene-Solr-6.x-Windows (32bit/jdk1.8.0_92) - Build # 304 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/304/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
ObjectTracker found 8 object(s) that were not released!!! 
[MockDirectoryWrapper, MDCAwareThreadPoolExecutor, TransactionLog, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, TransactionLog, 
MockDirectoryWrapper, MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 8 object(s) that were not 
released!!! [MockDirectoryWrapper, MDCAwareThreadPoolExecutor, TransactionLog, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, TransactionLog, 
MockDirectoryWrapper, MockDirectoryWrapper]
at __randomizedtesting.SeedInfo.seed([6A1867917CADF58A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:258)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
Could not remove the following files (in the order of attempts):
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data\tlog\tlog.000:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data\tlog\tlog.000:
 The process cannot access the file because it is being used by another 
process. 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data\tlog

C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_6A1867917CADF58A-001\tempDir-001\node2\testschemaapi_shard1_replica1\data

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+125) - Build # 17183 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17183/
Java: 64bit/jdk-9-ea+125 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.overseer.ZkStateReaderTest.testExternalCollectionWatchedNotWatched

Error Message:


Stack Trace:
java.lang.NullPointerException
at 
__randomizedtesting.SeedInfo.seed([13F674CB67AA16A8:184D85E736F3E001]:0)
at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testExternalCollectionWatchedNotWatched(ZkStateReaderTest.java:167)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:533)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)




Build Log:
[...truncated 12177 lines...]
   [junit4] Suite: org.apache.solr.cloud.overseer.ZkStateReaderTest
   [junit4]   2> Creating dataDir: 

[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_92) - Build # 5967 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/5967/
Java: 32bit/jdk1.8.0_92 -client -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestLocalFSCloudBackupRestore.test

Error Message:
expected: but was:

Stack Trace:
java.lang.AssertionError: expected: but was:
at 
__randomizedtesting.SeedInfo.seed([D9EA973593B7AB17:51BEA8EF3D4BC6EF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:209)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 12120 lines...]
   [junit4] Suite: 

[jira] [Commented] (SOLR-9252) Feature selection and logistic regression on text

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367151#comment-15367151
 ] 

Cao Manh Dat commented on SOLR-9252:


This is a good change. I think we should store the index of terms (from 0 -> 
number of terms) as well of example, so we can retrieve terms in sorted order.


> Feature selection and logistic regression on text
> -
>
> Key: SOLR-9252
> URL: https://issues.apache.org/jira/browse/SOLR-9252
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: SOLR-9252.patch, SOLR-9252.patch, enron1.zip
>
>
> SOLR-9186 come up with a challenges that for each iterative we have to 
> rebuild the tf-idf vector for each documents. It is costly computation if we 
> represent doc by a lot of terms. Features selection can help reducing the 
> computation.
> Due to its computational efficiency and simple interpretation, information 
> gain is one of the most popular feature selection methods. It is used to 
> measure the dependence between features and labels and calculates the 
> information gain between the i-th feature and the class labels 
> (http://www.jiliang.xyz/publication/feature_selection_for_classification.pdf).
> I confirmed that by running logistics regressions on enron mail dataset (in 
> which each email is represented by top 100 terms that have highest 
> information gain) and got the accuracy by 92% and precision by 82%.
> This ticket will create two new streaming expression. Both of them use the 
> same *parallel iterative framework* as SOLR-8492.
> {code}
> featuresSelection(collection1, q="*:*",  field="tv_text", outcome="out_i", 
> positiveLabel=1, numTerms=100)
> {code}
> featuresSelection will emit top terms that have highest information gain 
> scores. It can be combined with new tlogit stream.
> {code}
> tlogit(collection1, q="*:*",
>  featuresSelection(collection1, 
>   q="*:*",  
>   field="tv_text", 
>   outcome="out_i", 
>   positiveLabel=1, 
>   numTerms=100),
>  field="tv_text",
>  outcome="out_i",
>  maxIterations=100)
> {code}
> In the iteration n, the text logistics regression will emit nth model, and 
> compute the error of (n-1)th model. Because the error will be wrong if we 
> compute the error dynamically in each iteration. 
> In each iteration tlogit will change learning rate based on error of previous 
> iteration. It will increase the learning rate by 5% if error is going down 
> and It will decrease the learning rate by 50% if error is going up.
> This will support use cases such as building models for spam detection, 
> sentiment analysis and threat detection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9252) Feature selection and logistic regression on text

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367151#comment-15367151
 ] 

Cao Manh Dat edited comment on SOLR-9252 at 7/8/16 3:13 AM:


This is a good change. I think we should store the index of terms (from 0 -> 
number of terms) as well, so we can retrieve terms in sorted order.



was (Author: caomanhdat):
This is a good change. I think we should store the index of terms (from 0 -> 
number of terms) as well of example, so we can retrieve terms in sorted order.


> Feature selection and logistic regression on text
> -
>
> Key: SOLR-9252
> URL: https://issues.apache.org/jira/browse/SOLR-9252
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: SOLR-9252.patch, SOLR-9252.patch, enron1.zip
>
>
> SOLR-9186 come up with a challenges that for each iterative we have to 
> rebuild the tf-idf vector for each documents. It is costly computation if we 
> represent doc by a lot of terms. Features selection can help reducing the 
> computation.
> Due to its computational efficiency and simple interpretation, information 
> gain is one of the most popular feature selection methods. It is used to 
> measure the dependence between features and labels and calculates the 
> information gain between the i-th feature and the class labels 
> (http://www.jiliang.xyz/publication/feature_selection_for_classification.pdf).
> I confirmed that by running logistics regressions on enron mail dataset (in 
> which each email is represented by top 100 terms that have highest 
> information gain) and got the accuracy by 92% and precision by 82%.
> This ticket will create two new streaming expression. Both of them use the 
> same *parallel iterative framework* as SOLR-8492.
> {code}
> featuresSelection(collection1, q="*:*",  field="tv_text", outcome="out_i", 
> positiveLabel=1, numTerms=100)
> {code}
> featuresSelection will emit top terms that have highest information gain 
> scores. It can be combined with new tlogit stream.
> {code}
> tlogit(collection1, q="*:*",
>  featuresSelection(collection1, 
>   q="*:*",  
>   field="tv_text", 
>   outcome="out_i", 
>   positiveLabel=1, 
>   numTerms=100),
>  field="tv_text",
>  outcome="out_i",
>  maxIterations=100)
> {code}
> In the iteration n, the text logistics regression will emit nth model, and 
> compute the error of (n-1)th model. Because the error will be wrong if we 
> compute the error dynamically in each iteration. 
> In each iteration tlogit will change learning rate based on error of previous 
> iteration. It will increase the learning rate by 5% if error is going down 
> and It will decrease the learning rate by 50% if error is going up.
> This will support use cases such as building models for spam detection, 
> sentiment analysis and threat detection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_92) - Build # 1087 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1087/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelay

Error Message:
expected:<2> but was:<1>

Stack Trace:
java.lang.AssertionError: expected:<2> but was:<1>
at 
__randomizedtesting.SeedInfo.seed([BB98B328BE4AFE40:C40604ADD728D3CA]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdate(ZkStateReaderTest.java:130)
at 
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelay(ZkStateReaderTest.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 252 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/252/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  org.apache.solr.cloud.TestLocalFSCloudBackupRestore.test

Error Message:
Error from server at http://127.0.0.1:64046/solr: 'location' is not specified 
as a query parameter or as a default repository property or as a cluster 
property.

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:64046/solr: 'location' is not specified as a 
query parameter or as a default repository property or as a cluster property.
at 
__randomizedtesting.SeedInfo.seed([873664F4B23A05CD:F625B2E1CC66835]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:403)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:356)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1228)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:998)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:934)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testInvalidPath(AbstractCloudBackupRestoreTestCase.java:149)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)

Re: Re: Re: potential accuracy degradation due to approximation of document length in BM25 (and other similarities)

[Leo Boytsov]

Hi David,

thank you for picking it up. Now we are having a more meaningful discussion
regarding the "waste".

Leo,
> There may be confusion here as to where the space is wasted.  1 vs 8 bytes
> per doc on disk is peanuts, sure, but in RAM it is not and that is the
> concern.  AFAIK the norms are memory-mapped in, and we need to ensure it's
> trivial to know which offset to go to on disk based on a document id, which
> precludes compression but maybe you have ideas to improve that.
>

First, my understanding is that all essential parts of the Lucene index are
memory mapped, in particular, the inverted index (in the most common
scenario at least). Otherwise, the search performance is miserable. That
said, memory mapping a few extra bytes per document shouldn't make a
noticeable difference.

Also, judging by the code in the class Lucene53NormsProducer and a debug
session, Lucene only maps a compressed segment containing norm values.
Norms are stored using 1,2,4, or 8 bytes. They are uncompressed into an
8-byte long. This is probably on a per-slice basis.

Anyways, situations in which you will get more than 65536 words per
document are quite rare. Situations with documents having 4 billion words
(or more) are exotic. If you have such enormous documents, again, saving on
document normalization factors won't be your first priority. You would
probably think about the ways of splitting such a huge document containing
every possible keyword into something more manageable.

To sum up,  for 99.999% of the users squeezing normalization factors into a
single byte has absolutely no benefit. Memoization do seem to speed up
things a bit, but I suspect this may disappear with new generations of CPUs.


> To use your own norms encoding, see Codec.normsFormat.  (disclaimer: I
> haven't used this but I know where to look)
>

Ok, thanks.


>
> ~ David
>
> On Wed, Jul 6, 2016 at 5:31 PM Leo Boytsov  wrote:
>
> > Hi,
> >
> > for some reason I didn't get a reply from the mailing list directly, so I
> > have to send a new message. I appreciate if something can be fixed, so
> that
> > I get a reply as well.
> >
> > First of all, I don't buy the claim about the issue being well-known. I
> > would actually argue that nobody except a few Lucene devs know about it.
> > There is also a bug in Lucene's tutorial example. This needs to be fixed
> as
> > well.
> >
> > Neither do I find your arguments convincing. In particular, I don't think
> > that there is any serious waste of space. Please, see my detailed
> comments
> > below. Please, note that I definitely don't know all the internals well,
> so
> > I appreciate if you could explain them better.
> >
> > The downsides are documented and known. But I don't think you are
> >> fully documenting the tradeoffs here, by encoding up to a 64-bit long,
> >> you can use up to *8x more memory and disk space* than what lucene
> >> does by default, and that is per-field.
> >
> >
> > This is not true. First of all, the increase is only for the textual
> > fields. Simple fields like Integers don't use normalization factors. So,
> > there is no increase for them.
> >
> > In the worst case, you will have 7 extra bytes for a *text* field.
> > However, this is not an 8x increase.
> >
> > If you do *compress* the length of the text field, then its size will
> > depend on the size of the text field. For example, one extra byte will be
> > required for fields that contain
> > more than 256 words, two extra bytes for fields having more than 65536
> > words, and so on so forth. *Compared to the field sizes, a several byte*
> > increase is simply *laughable*.
> >
> > If Lucene saves normalization factor *without compression, *it should now
> > use 8 bytes already. So, storing the full document length won't make a
> > difference.
> >
> >
> >> So that is a real trap. Maybe
> >> throw an exception there instead if the boost != 1F (just don't
> >> support it), and add a guard for "supermassive" documents, so that at
> >> most only 16 bits are ever used instead of 64. The document need not
> >> really be massive, it can happen just from a strange analysis chain
> >> (n-grams etc) that you get large values here.
> >>
> >
> > As mentioned above, storing a few extra bytes for supermassive documents
> > doesn't affect the overall storage by more than a tiny fraction of a
> > percent.
> >
> >
> >>
> >> I have run comparisons in the past on standard collections to see what
> >> happens with this "feature"  and differences were very small. I think
> >> in practice people do far more damage by sharding their document
> >> collections but not using a distributed interchange of IDF, causing
> >> results from different shards to be incomparable :)
> >>
> >
> > Ok, this is not what I see on my data. I see* more than* a 10%
> > degradation. This is not peanuts. Do we want to re-run experiments on
> > standard collections? Don't forget that Lucene is now used as a baseline
> to
> > compare against. People claim 

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+125) - Build # 17182 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17182/
Java: 64bit/jdk-9-ea+125 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 36971 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3252 (and 2109 related) class file(s) for forbidden 
API invocations (in 1.94s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:347: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:508: 
Check for forbidden API calls failed, see log.

Total time: 62 minutes 18 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-master - Build # 537 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/537/

No tests ran.

Build Log:
[...truncated 40562 lines...]
prepare-release-no-sign:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist
 [copy] Copying 476 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/lucene
 [copy] Copying 245 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/solr
   [smoker] Java 1.8 
JAVA_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8
   [smoker] NOTE: output encoding is UTF-8
   [smoker] 
   [smoker] Load release URL 
"file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/dist/"...
   [smoker] 
   [smoker] Test Lucene...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.01 sec (16.8 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download lucene-7.0.0-src.tgz...
   [smoker] 29.8 MB in 0.03 sec (924.4 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-7.0.0.tgz...
   [smoker] 64.3 MB in 0.07 sec (937.4 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-7.0.0.zip...
   [smoker] 74.9 MB in 0.09 sec (874.3 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack lucene-7.0.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6022 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-7.0.0.zip...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6022 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-7.0.0-src.tgz...
   [smoker] make sure no JARs/WARs in src dist...
   [smoker] run "ant validate"
   [smoker] run tests w/ Java 8 and testArgs='-Dtests.slow=false'...
   [smoker] test demo with 1.8...
   [smoker]   got 222 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] generate javadocs w/ Java 8...
   [smoker] 
   [smoker] Crawl/parse...
   [smoker] 
   [smoker] Verify...
   [smoker]   confirm all releases have coverage in TestBackwardsCompatibility
   [smoker] find all past Lucene releases...
   [smoker] run TestBackwardsCompatibility..
   [smoker] success!
   [smoker] 
   [smoker] Test Solr...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.00 sec (35.1 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download solr-7.0.0-src.tgz...
   [smoker] 39.1 MB in 0.49 sec (80.0 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-7.0.0.tgz...
   [smoker] 137.2 MB in 1.29 sec (106.3 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-7.0.0.zip...
   [smoker] 145.9 MB in 1.19 sec (123.1 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack solr-7.0.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] unpack lucene-7.0.0.tgz...
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0/contrib/dataimporthandler-extras/lib/javax.mail-1.5.1.jar:
 it has javax.* classes
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0/contrib/dataimporthandler-extras/lib/activation-1.1.1.jar:
 it has javax.* classes
   [smoker] copying unpacked distribution for Java 8 ...
   [smoker] test solr example w/ Java 8...
   [smoker]   start Solr instance 
(log=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8/solr-example.log)...
   [smoker] No process found for Solr node running on port 8983
   [smoker]   Running techproducts example on port 8983 from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8
   [smoker] Creating Solr home directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/build/smokeTestRelease/tmp/unpack/solr-7.0.0-java8/example/techproducts/solr
   [smoker] 
   [smoker] Starting up Solr on port 8983 using command:
   [smoker] bin/solr start -p 8983 -s "example/techproducts/solr"
   [smoker] 
   [smoker] Waiting up to 30 seconds to see Solr running on port 8983 [|]  
 [/]   [-]   [\]   [|]   [/]   [-]   
[\]   [|]  

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367017#comment-15367017
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit b0386f0d643febdaed3257584eb06af08f5c in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b0386f0 ]

SOLR-9236: Don't use System.currentTimeMillis.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367016#comment-15367016
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit 91a9d96454c80f5f414170c4231c5b22fb094215 in lucene-solr's branch 
refs/heads/master from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=91a9d96 ]

SOLR-9236: Don't use System.currentTimeMillis.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 702 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/702/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:44698/yp_/f/c8n_1x3_lf_shard1_replica1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:44698/yp_/f/c8n_1x3_lf_shard1_replica1]
at 
__randomizedtesting.SeedInfo.seed([6988E10ACAB1A065:E1DCDED0644DCD9D]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:753)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1151)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1040)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:976)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:178)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366992#comment-15366992
 ] 

Steve Rowe commented on SOLR-9236:
--

[~markrmil...@gmail.com], did you see the following Jenkins precommit failure?:

{noformat}
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3254 (and 2112 related) class file(s) for forbidden 
API invocations (in 5.11s), 2 error(s).
{noformat}

> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 1264 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/1264/

4 tests failed.
FAILED:  org.apache.solr.cloud.CollectionStateFormat2Test.test

Error Message:
Could not find collection:.system

Stack Trace:
java.lang.AssertionError: Could not find collection:.system
at 
__randomizedtesting.SeedInfo.seed([7B0349932A95064B:F357764984696BB3]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at 
org.apache.solr.cloud.CollectionStateFormat2Test.testConfNameAndCollectionNameSame(CollectionStateFormat2Test.java:53)
at 
org.apache.solr.cloud.CollectionStateFormat2Test.test(CollectionStateFormat2Test.java:40)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Updated] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[Eungsop Yoo (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eungsop Yoo updated SOLR-9236:
--
Description: 
I need to index huge amount of logs, so I decide to use AutoAddReplica feature 
with only one replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance 

I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents 
during failover.
I found two reasons for the missing.

1. The leader replica does not replay any transaction logs. But when there is 
only one replica, it should be the leader.
So I made the leader replica replay the transaction logs when using 
AutoAddReplicas with on replica.

But the above fix did not resolve the problem.

2. As failover occurred, the transaction log directory had a deeper directory 
depth. Just like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log 
directory was changed during failover. 
So I made the transaction log directory not changed during failover.

After these fixes, AutoAddReplicas with one replica fails over well without 
losing any documents.


  was:
I need to index huge amount of logs, so I decide to use AutoAddReplica feature 
with only one replica.
When using AutoAddReplicas with one replica, some benefits are expected.
- no redundant data files for replicas
-- saving disk usage
- best indexing performance 

I expected that Solr fails over just like HBase.
The feature worked almost as it was expected, except for some missing documents 
during failover.
I found two regions for the missing.

1. The leader replica does not replay any transaction logs. But when there is 
only one replica, it should be the leader.
So I made the leader replica replay the transaction logs when using 
AutoAddReplicas with on replica.

But the above fix did not resolve the problem.

2. As failover occurred, the transaction log directory had a deeper directory 
depth. Just like this, tlog/tlog/tlog/...
The transaction log could not be replayed, because the transaction log 
directory was changed during failover. 
So I made the transaction log directory not changed during failover.

After these fixes, AutoAddReplicas with one replica fails over well without 
losing any documents.



> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two reasons for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch

New patch that passes the forbidden api checks.  I had to add one suppression 
as follows:
{code}
HttpServletResponse rspCloseShield = new HttpServletResponseWrapper(frsp) {
  @SuppressForbidden(reason = "Hadoop DelegationTokenAuthenticationFilter 
uses response writer, this" +
  "is providing a CloseShield on top of that")
  @Override
  public PrintWriter getWriter() throws IOException {
final PrintWriter pw = new PrintWriterWrapper(frsp.getWriter()) {
  @Override
  public void close() {};
};
return pw;
  }
};
{code}

I'm not 100% sure if the getWriter problem affects the hadoop usage, which is 
here:
https://github.com/apache/hadoop/blob/9d46a49c746b9e1ef552dbb10d1e22f87db68c76/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/delegation/web/DelegationTokenAuthenticationHandler.java#L280-L282

I can certainly submit a patch to hadoop (or add it to HADOOP-13346) to use 
OutputStream instead -- [~thetaphi]?

> Add Delegation Token Support to Solr
> 
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
>  Issue Type: New Feature
>  Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch, SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop 
> authentication filter.  Hadoop also has support for an authentication filter 
> that supports delegation tokens, which allow authenticated users the ability 
> to grab/renew/delete a token that can be used to bypass the normal 
> authentication path for a time.  This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access 
> to the user's kerberos credentials.  Instead, the job runner can grab a 
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can 
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more 
> privileged user can request a delegation token that can be passed to the less 
> privileged user.
> Note to self:
> In 
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
>  I made the following comment which I need to investigate further, since I 
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin 
> moving forward (I understand this is more a generic auth question than 
> kerberos specific). For example, in the latest version of the filter we are 
> using at Cloudera, we play around with the ServletContext in order to pass 
> information around 
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
>  Is there any way we can get the actual ServletContext in a plugin?{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_92) - Build # 1086 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1086/
Java: 64bit/jdk1.8.0_92 -XX:+UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 37097 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3254 (and 2112 related) class file(s) for forbidden 
API invocations (in 2.13s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:347: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:508: Check 
for forbidden API calls failed, see log.

Total time: 63 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 262 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/262/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 37003 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3254 (and 2112 related) class file(s) for forbidden 
API invocations (in 5.11s), 2 error(s).

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:740: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:117: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:347: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/common-build.xml:508: 
Check for forbidden API calls failed, see log.

Total time: 104 minutes 27 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr

[Gregory Chanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366917#comment-15366917
 ] 

Gregory Chanan commented on SOLR-9200:
--

CC [~anshumg], who worked on SOLR-7274.

> Add Delegation Token Support to Solr
> 
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
>  Issue Type: New Feature
>  Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop 
> authentication filter.  Hadoop also has support for an authentication filter 
> that supports delegation tokens, which allow authenticated users the ability 
> to grab/renew/delete a token that can be used to bypass the normal 
> authentication path for a time.  This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access 
> to the user's kerberos credentials.  Instead, the job runner can grab a 
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can 
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more 
> privileged user can request a delegation token that can be passed to the less 
> privileged user.
> Note to self:
> In 
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
>  I made the following comment which I need to investigate further, since I 
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin 
> moving forward (I understand this is more a generic auth question than 
> kerberos specific). For example, in the latest version of the filter we are 
> using at Cloudera, we play around with the ServletContext in order to pass 
> information around 
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
>  Is there any way we can get the actual ServletContext in a plugin?{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr

[Gregory Chanan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366913#comment-15366913
 ] 

Gregory Chanan commented on SOLR-9200:
--

Functionality:

1) This patch allows configuration of delegation tokens for the KerberosPlugin. 
 Existing
configurations will not change and will not support delegation tokens.

The configuration parameters are as follows:
||Key||Type||Default||Description||
|solr.kerberos.delegation.token.enabled|boolean|false|Set to true to enable 
delegation tokens
|solr.kerberos.delegation.token.kind|String|solr-dt|Type name of delegation 
tokens, most likely doesn't need to change
|solr.kerberos.delegation.token.validity|integer|36000|Lifetime in seconds for 
which delegation tokens are valid|
|solr.kerberos.delegation.token.signer.secret.provider|String|zookeeper|Where 
delegation token information is stored internally; if not "zookeeper" 
delegation tokens won't work across solr servers
|solr.kerberos.delegation.token.signer.secret.provider.zookeper.path|String|(chrooted
 path) + /token|Zookeeper location where secret provider information is stored
|solr.kerberos.delegation.token.secret.manager.znode.working.path|String|(chrooted
 path) + /zkdtsm|Zookeeper location where token information is stored

2) Includes solrj support for delegation tokens as follows:
  a) DelegationTokenRequest/DelegationTokenResponse can be used to 
get/cancel/renew delegation tokens
  b) HttpSolrClient.Builder now includes a "withDelegationToken" function for 
creating an HttpSolrClient
 that uses a delegation token to authenticate
  Note that hadoop's delegation token responses are in json map format, so 
there is a new ResponseParser
  for that in DelegationTokenResponse; I couldn't find an existing response 
parser that worked
  
Issues / Workarounds:

3) AuthenticationPlugin has an incompatible change that means this should only 
go into the next major version.
Basically:
{code}void doAuthenticate{code}
changed to:
{code}boolean doAuthenticate{code}
This is to support cases where authentication succeeds, but solr itself 
shouldn't process the request; e.g.
in the delegation token management operations (get, cancel, renew).  The 
boolean, if false, indicates a short
circuit out of the rest of the solr dispatch logic.

4) DelegationTokenKerberosFilter includes a workaround for null query strings.  
The versions of
hadoop / httpclient that we use don't work with null HttpServletRequest query 
strings, which the jetty
version we use can provide.  This can be solved by using HTTPCLIENT-1746 (not 
released yet) or HADOOP-12767
(not released in a stable version).

5) hadoop's delegation token code writes to a closed PrintWriter; this doesn't 
seem to be a problem for the
version of jetty that hadoop uses, but it causes an issue with our version.  I 
filed HADOOP-13346 to fix that,
until then, I wrote a PrintWriterWrapper that ignores closes.

6) The hadoop zookeeper delegation token code uses Curator rather than 
SolrZkClient; the conversion
from SolrZkClient is messy in a few places:
  a) We use the ZkController.ZkClient's ACL Provider for the delegation tokens 
in ZK, but it's not obvious this
is what you'd actually want to use.  For example, it may be reasonable to 
have most solr znodes be readable
(because clients read e.g. clusterstate.json), but you probably don't want 
the delegation token information
to be readable by anyone outside "solr".  I haven't checked recently, but I 
don't think we provide any
built in ACLProviders that would do something reasonable here in the 
general case.  Basically, it's easy to
get this wrong and leak security information.
  b) Getting credentials information to curator also isn't great.  Again, we 
use ZkController.ZkClient's credentials
(at AuthenticationPlugin.init) time, but given the 
"setZkCredentialsToAddAutomatically" function, these could
in theory change.  I haven't looked into changing that into a builder so 
it's guaranteed not to change.
  c) Retrying logic is handled completely differently.  In theory, you'd like 
the curator logic to follow ZkController.ZkClient's
 ZkClientConnectionStrategy logic, but it's not clear how to implement 
this.  Instead, we just use curator's ExponentialBackoffRetry.

> Add Delegation Token Support to Solr
> 
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
>  Issue Type: New Feature
>  Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop 
> authentication filter.  Hadoop also has support for an authentication filter 
> that supports delegation tokens, which allow authenticated users the ability 
> to 

[JENKINS] Lucene-Solr-Tests-6.x - Build # 322 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/322/

3 tests failed.
FAILED:  org.apache.solr.update.HardAutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([A366FD873484FE74:19B492FFB7AA1061]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:783)
at 
org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:q=id:529=standard=0=20=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:776)
... 40 more


FAILED:  
org.apache.solr.cloud.overseer.ZkStateReaderTest.testStateFormatUpdateWithTimeDelayLazy

Error Message:
expected:<2> but was:<1>

Stack 

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+125) - Build # 17181 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17181/
Java: 64bit/jdk-9-ea+125 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 36974 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3252 (and 2109 related) class file(s) for forbidden 
API invocations (in 1.89s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:347: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:508: 
Check for forbidden API calls failed, see log.

Total time: 65 minutes 49 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9088) solr.schema.TestManagedSchemaAPI.test failures ([doc=2] unknown field 'myNewField1')

[Varun Thacker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker resolved SOLR-9088.
-
Resolution: Fixed
  Assignee: Varun Thacker

> solr.schema.TestManagedSchemaAPI.test failures ([doc=2] unknown field 
> 'myNewField1')
> 
>
> Key: SOLR-9088
> URL: https://issues.apache.org/jira/browse/SOLR-9088
> Project: Solr
>  Issue Type: Test
>Reporter: Christine Poerschke
>Assignee: Varun Thacker
>Priority: Minor
> Fix For: 6.2
>
> Attachments: SOLR-9088.patch, SOLR-9088.patch
>
>
> e.g.
> http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3256/
> http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/588/
> http://jenkins.thetaphi.de/job/Lucene-Solr-5.5-Linux/266/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9088) solr.schema.TestManagedSchemaAPI.test failures ([doc=2] unknown field 'myNewField1')

[Varun Thacker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker updated SOLR-9088:

Fix Version/s: 6.2

> solr.schema.TestManagedSchemaAPI.test failures ([doc=2] unknown field 
> 'myNewField1')
> 
>
> Key: SOLR-9088
> URL: https://issues.apache.org/jira/browse/SOLR-9088
> Project: Solr
>  Issue Type: Test
>Reporter: Christine Poerschke
>Priority: Minor
> Fix For: 6.2
>
> Attachments: SOLR-9088.patch, SOLR-9088.patch
>
>
> e.g.
> http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3256/
> http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/588/
> http://jenkins.thetaphi.de/job/Lucene-Solr-5.5-Linux/266/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-MAVEN] Lucene-Solr-Maven-master #1802: POMs out of sync

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Maven-master/1802/

No tests ran.

Build Log:
[...truncated 45482 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/build.xml:763: The 
following error occurred while executing this line:
: Java returned: 1

Total time: 32 minutes 34 seconds
Build step 'Invoke Ant' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3394 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3394/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[http://127.0.0.1:65321/forceleader_test_collection_shard1_replica1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[http://127.0.0.1:65321/forceleader_test_collection_shard1_replica1]
at 
__randomizedtesting.SeedInfo.seed([E2EC1805894FB65D:47B2CC5B0CD4F3C]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:753)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1151)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1040)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:976)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
at 
org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
at 
org.apache.solr.cloud.ForceLeaderTest.assertSendDocFails(ForceLeaderTest.java:315)
at 
org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[JENKINS-EA] Lucene-Solr-6.x-Linux (32bit/jdk-9-ea+125) - Build # 1085 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1085/
Java: 32bit/jdk-9-ea+125 -server -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 37089 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3254 (and 2113 related) class file(s) for forbidden 
API invocations (in 2.20s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:347: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:508: Check 
for forbidden API calls failed, see log.

Total time: 78 minutes 55 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-6.x - Build # 105 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-6.x/105/

No tests ran.

Build Log:
[...truncated 40568 lines...]
prepare-release-no-sign:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist
 [copy] Copying 476 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/lucene
 [copy] Copying 245 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/solr
   [smoker] Java 1.8 
JAVA_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/latest1.8
   [smoker] NOTE: output encoding is UTF-8
   [smoker] 
   [smoker] Load release URL 
"file:/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/dist/"...
   [smoker] 
   [smoker] Test Lucene...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.01 sec (14.0 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download lucene-6.2.0-src.tgz...
   [smoker] 29.9 MB in 0.04 sec (818.6 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-6.2.0.tgz...
   [smoker] 64.4 MB in 0.08 sec (832.1 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download lucene-6.2.0.zip...
   [smoker] 75.0 MB in 0.09 sec (795.5 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack lucene-6.2.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6036 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-6.2.0.zip...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] test demo with 1.8...
   [smoker]   got 6036 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] check Lucene's javadoc JAR
   [smoker]   unpack lucene-6.2.0-src.tgz...
   [smoker] make sure no JARs/WARs in src dist...
   [smoker] run "ant validate"
   [smoker] run tests w/ Java 8 and testArgs='-Dtests.slow=false'...
   [smoker] test demo with 1.8...
   [smoker]   got 224 hits for query "lucene"
   [smoker] checkindex with 1.8...
   [smoker] generate javadocs w/ Java 8...
   [smoker] 
   [smoker] Crawl/parse...
   [smoker] 
   [smoker] Verify...
   [smoker]   confirm all releases have coverage in TestBackwardsCompatibility
   [smoker] find all past Lucene releases...
   [smoker] run TestBackwardsCompatibility..
   [smoker] success!
   [smoker] 
   [smoker] Test Solr...
   [smoker]   test basics...
   [smoker]   get KEYS
   [smoker] 0.2 MB in 0.00 sec (33.3 MB/sec)
   [smoker]   check changes HTML...
   [smoker]   download solr-6.2.0-src.tgz...
   [smoker] 39.2 MB in 0.57 sec (68.6 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-6.2.0.tgz...
   [smoker] 137.2 MB in 1.51 sec (90.6 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   download solr-6.2.0.zip...
   [smoker] 145.9 MB in 1.85 sec (78.8 MB/sec)
   [smoker] verify md5/sha1 digests
   [smoker]   unpack solr-6.2.0.tgz...
   [smoker] verify JAR metadata/identity/no javax.* or java.* classes...
   [smoker] unpack lucene-6.2.0.tgz...
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.2.0/contrib/dataimporthandler-extras/lib/javax.mail-1.5.1.jar:
 it has javax.* classes
   [smoker]   **WARNING**: skipping check of 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.2.0/contrib/dataimporthandler-extras/lib/activation-1.1.1.jar:
 it has javax.* classes
   [smoker] copying unpacked distribution for Java 8 ...
   [smoker] test solr example w/ Java 8...
   [smoker]   start Solr instance 
(log=/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.2.0-java8/solr-example.log)...
   [smoker] No process found for Solr node running on port 8983
   [smoker]   Running techproducts example on port 8983 from 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.2.0-java8
   [smoker] Creating Solr home directory 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-6.x/lucene/build/smokeTestRelease/tmp/unpack/solr-6.2.0-java8/example/techproducts/solr
   [smoker] 
   [smoker] Starting up Solr on port 8983 using command:
   [smoker] bin/solr start -p 8983 -s "example/techproducts/solr"
   [smoker] 
   [smoker] Waiting up to 30 seconds to see Solr running on port 8983 [|]  
 [/]   [-]   [\]   [|]   [/]   [-]   
[\]   [|]   [/]   [-]  
   

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366706#comment-15366706
 ] 

Mark Miller commented on SOLR-7280:
---

bq. Unless we don't let users create shards with that many replicas or 
something.

Although even that is probably not enough unless we only load one shard at a 
time.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366705#comment-15366705
 ] 

Mark Miller commented on SOLR-7280:
---

bq. That said, I think the default should be quite high in the cloud case so we 
don't change the current behavior and let situations like I'm seeing deal with 
configuring this. I think it defaults to 8 currently, perhaps 100 (or 
unlimited) instead in cloud mode?

I think it's trappy, whatever the default is. Unless we don't let users create 
shards with that many replicas or something.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366703#comment-15366703
 ] 

Mark Miller commented on SOLR-7280:
---

bq. I'll emphasize though that the current code (without this patch) can 
prevent a cluster from coming up at all.

The current code is a fix to a bug that existed when we did something like this 
before :) I don't want to reintroduce the bug. We didn't, and as far as I know 
still don't, support tons of cores very well. As we move to do that, we don't 
want to reintroduce bugs though.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366690#comment-15366690
 ] 

Erick Erickson edited comment on SOLR-7280 at 7/7/16 8:03 PM:
--

bq: I don't think it takes a weird topology - just more replicas than thread to 
load them in a shard.

OK, I think I see what you're saying. You're talking about a "deep" topology, 
i.e. one with many replicas on a particular shard on a particular instance and 
I was looking at a "wide" topology, many collections per instance but each 
shard had only a few replicas. I've seen both in the field as I'm sure you 
have

How much of both situations would be handled by creating an ordered list of all 
replicas that were leaders and loading those first then loading an ordered list 
of all replicas that weren't labeled as leader? There's still the case of a 
zillion leaders on a single instance, so some heuristic like you suggest seems 
to be in order.

I'll emphasize though that the current code (without this patch) can prevent a 
cluster from coming up at _all_. With this patch the cluster at least comes up, 
albeit slowly if the leaderVoteWait comes into play. Bumping the number of 
threads to > the max replicas for a shard can handle the case you mentioned 
while keeping it "reasonable" can deal with the one I'm seeing.

That said, I think the default should be quite high in the cloud case so we 
don't change the current behavior and let situations like I'm seeing deal with 
configuring this. I think it defaults to 8 currently, perhaps 100 (or 
unlimited) instead in cloud mode?

How much of all of the above makes this patch "good enough for now" with 
perhaps follow-ons on more sophisticated approaches?


was (Author: erickerickson):
bq: I don't think it takes a weird topology - just more replicas than thread to 
load them in a shard.

OK, I think I see what you're saying. You're talking about a "deep" topology, 
i.e. one with many replicas on a particular shard on a particular instance and 
I was looking at a "wide" topology, many collections per instance but each 
shard had only a few replicas. I've seen both in the field as I'm sure you 
have

How much of both situations would be handled by creating an ordered list of all 
replicas that were leaders and loading those first then loading an ordered list 
of all replicas that weren't labeled as leader? There's still the case of a 
zillion leaders on a single instance, so some heuristic like you suggest seems 
to be in order.

I'll emphasize though that the current code (without this patch) can prevent a 
cluster from coming up at _all_. With this patch the cluster at least comes up, 
albeit slowly if the leaderVoteWait comes into play. Bumping the number of 
threads can to > the max replicas for a shard can handle the case you mentioned 
while keeping it "reasonable" can deal with the one I'm seeing.

That said, I think the default should be quite high in the cloud case so we 
don't change the current behavior and let situations like I'm seeing deal with 
configuring this. I think it defaults to 8 currently, perhaps 100 (or 
unlimited) instead in cloud mode?

How much of all of the above makes this patch "good enough for now" with 
perhaps follow-ons on more sophisticated approaches?

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366690#comment-15366690
 ] 

Erick Erickson commented on SOLR-7280:
--

bq: I don't think it takes a weird topology - just more replicas than thread to 
load them in a shard.

OK, I think I see what you're saying. You're talking about a "deep" topology, 
i.e. one with many replicas on a particular shard on a particular instance and 
I was looking at a "wide" topology, many collections per instance but each 
shard had only a few replicas. I've seen both in the field as I'm sure you 
have

How much of both situations would be handled by creating an ordered list of all 
replicas that were leaders and loading those first then loading an ordered list 
of all replicas that weren't labeled as leader? There's still the case of a 
zillion leaders on a single instance, so some heuristic like you suggest seems 
to be in order.

I'll emphasize though that the current code (without this patch) can prevent a 
cluster from coming up at _all_. With this patch the cluster at least comes up, 
albeit slowly if the leaderVoteWait comes into play. Bumping the number of 
threads can to > the max replicas for a shard can handle the case you mentioned 
while keeping it "reasonable" can deal with the one I'm seeing.

That said, I think the default should be quite high in the cloud case so we 
don't change the current behavior and let situations like I'm seeing deal with 
configuring this. I think it defaults to 8 currently, perhaps 100 (or 
unlimited) instead in cloud mode?

How much of all of the above makes this patch "good enough for now" with 
perhaps follow-ons on more sophisticated approaches?

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Windows (64bit/jdk1.8.0_92) - Build # 303 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/303/
Java: 64bit/jdk1.8.0_92 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.cloud.TestLocalFSCloudBackupRestore.test

Error Message:
Error from server at http://127.0.0.1:51900/solr: The backup directory already 
exists: 
file:///C:/Users/jenkins/workspace/Lucene-Solr-6.x-Windows/solr/build/solr-core/test/J1/temp/solr.cloud.TestLocalFSCloudBackupRestore_F8C09989FDAB73EC-001/tempDir-002/mytestbackup/

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:51900/solr: The backup directory already 
exists: 
file:///C:/Users/jenkins/workspace/Lucene-Solr-6.x-Windows/solr/build/solr-core/test/J1/temp/solr.cloud.TestLocalFSCloudBackupRestore_F8C09989FDAB73EC-001/tempDir-002/mytestbackup/
at 
__randomizedtesting.SeedInfo.seed([F8C09989FDAB73EC:7094A65353571E14]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:590)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:403)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:356)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1228)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:998)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:934)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:207)
at 
org.apache.solr.cloud.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+125) - Build # 17180 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17180/
Java: 64bit/jdk-9-ea+125 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 36977 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3252 (and 2109 related) class file(s) for forbidden 
API invocations (in 1.84s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:347: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:508: 
Check for forbidden API calls failed, see log.

Total time: 66 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7495) Unexpected docvalues type NUMERIC when grouping by a int facet

[Anton Khoff (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366626#comment-15366626
 ] 

Anton Khoff commented on SOLR-7495:
---

And what about 6.1?

> Unexpected docvalues type NUMERIC when grouping by a int facet
> --
>
> Key: SOLR-7495
> URL: https://issues.apache.org/jira/browse/SOLR-7495
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.0, 5.1, 5.2, 5.3
>Reporter: Fabio Batista da Silva
> Attachments: SOLR-7495.patch, SOLR-7495.patch
>
>
> Hey All,
> After upgrading from solr 4.10 to 5.1 with solr could
> I'm getting a IllegalStateException when i try to facet a int field.
> IllegalStateException: unexpected docvalues type NUMERIC for field 'year' 
> (expected=SORTED). Use UninvertingReader or index with docvalues.
> schema.xml
> {code}
> 
> 
> 
> 
> 
> 
>  multiValued="false" required="true"/>
>  multiValued="false" required="true"/>
> 
> 
>  stored="true"/>
> 
> 
> 
>  />
>  sortMissingLast="true"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  precisionStep="0" positionIncrementGap="0"/>
>  positionIncrementGap="0"/>
>  positionIncrementGap="100">
> 
> 
>  words="stopwords.txt" />
> 
>  maxGramSize="15"/>
> 
> 
> 
>  words="stopwords.txt" />
>  synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> 
> 
> 
>  positionIncrementGap="100">
> 
> 
>  words="stopwords.txt" />
> 
>  maxGramSize="15"/>
> 
> 
> 
>  words="stopwords.txt" />
>  synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
> 
> 
> 
>  class="solr.SpatialRecursivePrefixTreeFieldType" geo="true" 
> distErrPct="0.025" maxDistErr="0.09" units="degrees" />
> 
> id
> name
> 
> 
> {code}
> query :
> {code}
> http://solr.dev:8983/solr/my_collection/select?wt=json=id=index_type:foobar=true=year_make_model=true=true=year
> {code}
> Exception :
> {code}
> ull:org.apache.solr.common.SolrException: Exception during facet.field: year
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:627)
> at org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:612)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:566)
> at 
> org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:637)
> at 
> org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:280)
> at 
> org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:106)
> at 
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:222)
> at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:143)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1984)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:829)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:446)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> at 
> 

[JENKINS-MAVEN] Lucene-Solr-Maven-6.x #108: POMs out of sync

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Maven-6.x/108/

No tests ran.

Build Log:
[...truncated 45492 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-6.x/build.xml:763: The 
following error occurred while executing this line:
: Java returned: 1

Total time: 33 minutes 58 seconds
Build step 'Invoke Ant' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9283) Documentation on using preferredNodes-ImplicitSnitch for Multi Data Center scenario

[Susheel Kumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366619#comment-15366619
 ] 

Susheel Kumar commented on SOLR-9283:
-

Thank you Noble for your response. Let me know if anyone is working on
completing the patch or I can give a shot on the patch if you can provide
some direction/design.

Thanks,
Susheel




> Documentation on using preferredNodes-ImplicitSnitch for Multi Data Center 
> scenario
> ---
>
> Key: SOLR-9283
> URL: https://issues.apache.org/jira/browse/SOLR-9283
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.0
>Reporter: Susheel Kumar
>Priority: Blocker
>  Labels: documentation
>
> SOLR-8146 and SOLR-8522 has been worked on to allow SolrJ CloudSolrClient to 
> have preferred replica for query/read more specifically for multiple Data 
> Center's scenario but there is no/unclear documentation on how exactly this 
> feature can be used and what steps needs to be taken care at SolrJ client 
> side and part of collection configuration/state.
> I am offering help to create the required documentation if little more 
> details can be provided.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366618#comment-15366618
 ] 

Mark Miller commented on SOLR-7280:
---

One strategy might be to only require a certain number of replicas participate 
in an election to elect a leader and then tie that to the number of threads 
used to load replicas and then start replicas in some intelligent, per shard 
manner.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366614#comment-15366614
 ] 

Mark Miller commented on SOLR-7280:
---

bq. Sure, with weird enough topology

I don't think it takes a weird topology - just more replicas than thread to 
load them in a shard.

The current behavior is trappy if you want to try and load tons and tons of 
cores. The change in behavior is trappy in general.

I think we always anticipated a better strategy to deal with this, but I don't 
think this does it very well.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 113 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/113/

2 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:40981

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:40981
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:601)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:259)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:248)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.makeRequest(CollectionsAPIDistributedZkTest.java:399)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.testErrorHandling(CollectionsAPIDistributedZkTest.java:416)
at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test(CollectionsAPIDistributedZkTest.java:179)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  

[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_92) - Build # 5966 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/5966/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseSerialGC

2 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[http://127.0.0.1:54600/kwz/forceleader_test_collection_shard1_replica1]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this 
request:[http://127.0.0.1:54600/kwz/forceleader_test_collection_shard1_replica1]
at 
__randomizedtesting.SeedInfo.seed([B29FD80DBDE8F8A4:5408ECCD846A01C5]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:753)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1151)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1040)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:976)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:753)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.sendDocsWithRetry(AbstractFullDistribZkTestBase.java:741)
at 
org.apache.solr.cloud.ForceLeaderTest.sendDoc(ForceLeaderTest.java:424)
at 
org.apache.solr.cloud.ForceLeaderTest.assertSendDocFails(ForceLeaderTest.java:315)
at 
org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:110)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366563#comment-15366563
 ] 

Erick Erickson commented on SOLR-7280:
--

The symptom is that  OOM errors "unable to create native thread" happens, 
resulting in replicas that never come up, sometimes never ending recovery 
cycles etc. etc. etc. Decreasing Xss or increasing Xmx  doesn't help (maybe 
some other settings?). In my testing  with 400 replicas in a JVM, something on 
the order of 1K temporary threads were spun up when I tried to start the JVM. 
More correctly that many threads were running, the rest (number unknown) never 
started at all.

What the default should be is certainly debatable. The curious thing was that 
at no time did the JVM memory appear to be stressed (using jconsole to 
spot-check, not rigorous at all)

I've assumed that by ordering the replicas to come up based on what collection 
they belong to, they won't get stuck waiting for a leader election just because 
the ordering on instance1 happened to try to bring up collection1 then 
collection2 whereas instance2 tried to bring them up in reverse order. More 
like skip-lists in terms of waiting...

Sure, with weird enough topology instance1 could have a long queue to get 
through before getting to collectionX whereas instance N could start with 
collectionX and have to go through leader vote wait timeouts, but that's way 
better than having a cluster that won't start at all.

And when I tested starting 3 of my 4 JVMs, it was indeed painfully slow waiting 
for leader vote wait timeouts. But the cluster came up.

Using 3 coreLoadThreads and starting all my JVMs at once took just a few 
minutes.

And the painfully trappy behavior without this patch is that I can create all 
my collections just fine, but then I can't restart the cluster successfully.



> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9207) PeerSync recovery fails if number of updates requested is high

[Pushkar Raste (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9207?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushkar Raste updated SOLR-9207:

Summary: PeerSync recovery fails if number of updates requested is high  
(was: PeerSync recovery failes if number of updates requested is high)

> PeerSync recovery fails if number of updates requested is high
> --
>
> Key: SOLR-9207
> URL: https://issues.apache.org/jira/browse/SOLR-9207
> Project: Solr
>  Issue Type: Bug
>Affects Versions: 5.1, 6.0
>Reporter: Pushkar Raste
>Assignee: Shalin Shekhar Mangar
>Priority: Minor
> Fix For: 6.2, master (7.0)
>
> Attachments: SOLR-9207.patch, SOLR-9207.patch, SOLR-9207.patch_updated
>
>
> {{PeerSync}} recovery fails if we request more than ~99K updates. 
> If update solrconfig to retain more {{tlogs}} to leverage 
> https://issues.apache.org/jira/browse/SOLR-6359
> During out testing we found out that recovery using {{PeerSync}} fails if we 
> ask for more than ~99K updates, with following error
> {code}
>  WARN  PeerSync [RecoveryThread] - PeerSync: core=hold_shard1 url=
> exception talking to , failed
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: 
> Expected mime type application/octet-stream but got application/xml. 
> 
> 
> application/x-www-form-urlencoded content 
> length (4761994 bytes) exceeds upload limit of 2048 KB t name="code">400
> 
> {code}
> We arrived at ~99K with following match
> * max_version_number = Long.MAX_VALUE = 9223372036854775807  
> * bytes per version number =  20 (on the wire as POST request sends version 
> number as string)
> * additional bytes for separator ,
> * max_versions_in_single_request = 2MB/21 = ~99864
> I could think of 2 ways to fix it
> 1. Ask for about updates in chunks of 90K inside {{PeerSync.requestUpdates()}}
> 2. Use application/octet-stream encoding 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9241) Rebalance API for SolrCloud

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366531#comment-15366531
 ] 

Noble Paul commented on SOLR-9241:
--

This issue can be split into multiple subtasks and we can attack them one by 
one. One huge patch is very difficult to push through

> Rebalance API for SolrCloud
> ---
>
> Key: SOLR-9241
> URL: https://issues.apache.org/jira/browse/SOLR-9241
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Affects Versions: 4.6.1
> Environment: Ubuntu, Mac OsX
>Reporter: Nitin Sharma
>  Labels: Cluster, SolrCloud
> Fix For: 4.6.1
>
> Attachments: rebalance.diff
>
>   Original Estimate: 2,016h
>  Remaining Estimate: 2,016h
>
> This is the v1 of the patch for Solrcloud Rebalance api (as described in 
> http://engineering.bloomreach.com/solrcloud-rebalance-api/) , built at 
> Bloomreach by Nitin Sharma and Suruchi Shah. The goal of the API  is to 
> provide a zero downtime mechanism to perform data manipulation and  efficient 
> core allocation in solrcloud. This API was envisioned to be the base layer 
> that enables Solrcloud to be an auto scaling platform. (and work in unison 
> with other complementing monitoring and scaling features).
> Patch Status:
> ===
> The patch is work in progress and incremental. We have done a few rounds of 
> code clean up. We wanted to get the patch going first to get initial feed 
> back.  We will continue to work on making it more open source friendly and 
> easily testable.
>  Deployment Status:
> 
> The platform is deployed in production at bloomreach and has been battle 
> tested for large scale load. (millions of documents and hundreds of 
> collections).
>  Internals:
> =
> The internals of the API and performance : 
> http://engineering.bloomreach.com/solrcloud-rebalance-api/
> It is built on top of the admin collections API as an action (with various 
> flavors). At a high level, the rebalance api provides 2 constructs:
> Scaling Strategy:  Decides how to move the data.  Every flavor has multiple 
> options which can be reviewed in the api spec.
> Re-distribute  - Move around data in the cluster based on capacity/allocation.
> Auto Shard  - Dynamically shard a collection to any size.
> Smart Merge - Distributed Mode - Helps merging data from a larger shard setup 
> into smaller one.  (the source should be divisible by destination)
> Scale up -  Add replicas on the fly
> Scale Down - Remove replicas on the fly
> Allocation Strategy:  Decides where to put the data.  (Nodes with least 
> cores, Nodes that do not have this collection etc). Custom implementations 
> can be built on top as well. One other example is Availability Zone aware. 
> Distribute data such that every replica is placed on different availability 
> zone to support HA.
>  Detailed API Spec:
> 
>   https://github.com/bloomreach/solrcloud-rebalance-api
>  Contributors:
> =
>   Nitin Sharma
>   Suruchi Shah
>  Questions/Comments:
> =
>   You can reach me at nitin.sha...@bloomreach.com



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9188) BlockUnknown property makes inter-node communication impossible

[Alex D (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366524#comment-15366524
 ] 

Alex D commented on SOLR-9188:
--

On my development workstation (windows) I was able to workaround the issue by 
editing solr.in.cmd and changing SOLR_HOST to include a username & password. 
e.g.: 
set SOLR_HOST=username:passw...@mysolrhost.com

> BlockUnknown property makes inter-node communication impossible
> ---
>
> Key: SOLR-9188
> URL: https://issues.apache.org/jira/browse/SOLR-9188
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 6.0
>Reporter: Piotr Tempes
>
> When I setup my solr cloud without blockUnknown property it works as 
> expected. When I want to block non authenticated requests I get following 
> stacktrace during startup:
> {code}
> 2016-06-06 05:49:16.360 INFO  
> (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr 
> x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa 
> s:shard1 r:core_node1 x:testowa_shard1_replica3] 
> o.a.s.c.ShardLeaderElectionContext Enough replicas found to continue.
> 2016-06-06 05:49:16.360 INFO  
> (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr 
> x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa 
> s:shard1 r:core_node1 x:testowa_shard1_replica3] 
> o.a.s.c.ShardLeaderElectionContext I may be the new leader - try and sync
> 2016-06-06 05:49:16.361 INFO  
> (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr 
> x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa 
> s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.c.SyncStrategy Sync 
> replicas to https://172.30.92.66:8983/solr/testowa_shard1_replica3/
> 2016-06-06 05:49:16.364 INFO  
> (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr 
> x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa 
> s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: 
> core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr START 
> replicas=[https://172.30.182.43:8983/solr/testowa_shard1_replica1/, 
> https://172.30.182.44:8983/solr/testowa_shard1_replica2/] nUpdates=100
> 2016-06-06 05:49:16.365 INFO  
> (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_000150) 
> [   ] o.a.s.c.o.ReplicaMutator Update state numShards=1 message={
>   "core":"retailers_shard1_replica1",
>   "core_node_name":"core_node1",
>   "roles":null,
>   "base_url":"https://172.30.182.44:8983/solr;,
>   "node_name":"172.30.182.44:8983_solr",
>   "numShards":"1",
>   "state":"active",
>   "shard":"shard1",
>   "collection":"retailers",
>   "operation":"state"}
> 2016-06-06 05:49:16.481 INFO  
> (OverseerStateUpdate-240112591792046141-172.30.92.66:8983_solr-n_000150) 
> [   ] o.a.s.c.o.ZkStateWriter going to update_collection 
> /collections/retailers/state.json version: 89
> 2016-06-06 05:49:17.758 WARN  
> (coreZkRegister-1-thread-1-processing-n:172.30.92.66:8983_solr 
> x:testowa_shard1_replica3 s:shard1 c:testowa r:core_node1) [c:testowa 
> s:shard1 r:core_node1 x:testowa_shard1_replica3] o.a.s.u.PeerSync PeerSync: 
> core=testowa_shard1_replica3 url=https://172.30.92.66:8983/solr  exception 
> talking to https://172.30.182.44:8983/solr/testowa_shard1_replica2/, failed
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
> from server at https://172.30.182.44:8983/solr/testowa_shard1_replica2: 
> Expected mime type application/octet-stream but got text/html. 
> 
> 
> Error 401 Unauthorized request, Response code: 401
> 
> HTTP ERROR 401
> Problem accessing /solr/testowa_shard1_replica2/get. Reason:
> Unauthorized request, Response code: 401
> 
> 
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:545)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
>   at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
>   at 
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:198)
>   at 
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:163)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:277)
>   at 
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
>   at java.util.concurrent.FutureTask.run(FutureTask.java:277)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
>   at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$$Lambda$3.C403EBB0.run(Unknown
>  Source)
>   at 
> 

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_92) - Build # 1084 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1084/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 37085 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:317)
[forbidden-apis] Forbidden method invocation: 
java.lang.System#currentTimeMillis() [Use RTimer/TimeOut/System.nanoTime for 
time comparisons, and `new Date()` output/debugging/stats of timestamps. If for 
some miscellaneous reason, you absolutely need to use this, use a 
SuppressForbidden.]
[forbidden-apis]   in org.apache.solr.cloud.SharedFSAutoReplicaFailoverTest 
(SharedFSAutoReplicaFailoverTest.java:321)
[forbidden-apis] Scanned 3254 (and 2112 related) class file(s) for forbidden 
API invocations (in 2.55s), 2 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:347: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:508: Check 
for forbidden API calls failed, see log.

Total time: 68 minutes 15 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8297) Allow join query over 2 sharded collections: enhance functionality and exception handling

[Shikha Somani (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366512#comment-15366512
 ] 

Shikha Somani commented on SOLR-8297:
-

Please review these changes and let me know your thoughts on it. This issue is 
blocking our upgrade to Solr 5.x as it is directly impacting join functionality.

Appreciate your quick response on this.

> Allow join query over 2 sharded collections: enhance functionality and 
> exception handling
> -
>
> Key: SOLR-8297
> URL: https://issues.apache.org/jira/browse/SOLR-8297
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Affects Versions: 5.3
>Reporter: Paul Blanchaert
>
> Enhancement based on SOLR-4905. New Jira issue raised as suggested by Mikhail 
> Khludnev.
> A) exception handling:
> The exception "SolrCloud join: multiple shards not yet supported" thrown in 
> the function findLocalReplicaForFromIndex of JoinQParserPlugin is not 
> triggered correctly: In my use-case, I've a join on a facet.query and when my 
> results are only found in 1 shard and the facet.query with the join is 
> querying the last replica of the last slice, then the exception is not thrown.
> I believe it's better to verify the nr of slices when we want to verify the  
> "multiple shards not yet supported" exception (so exception is thrown when 
> zkController.getClusterState().getSlices(fromIndex).size()>1).
> B) functional enhancement:
> I would expect that there is no problem to perform a cross-core join over 
> sharded collections when the following conditions are met:
> 1) both collections are sharded with the same replicationFactor and numShards
> 2) router.field of the collections is set to the same "key-field" (collection 
> of "fromindex" has router.field = "from" field and collection joined to has 
> router.field = "to" field)
> The router.field setup ensures that documents with the same "key-field" are 
> routed to the same node. 
> So the combination based on the "key-field" should always be available within 
> the same node.
> From a user perspective, I believe these assumptions seem to be a "normal" 
> use-case in the cross-core join in SolrCloud.
> Hope this helps



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9200) Add Delegation Token Support to Solr

[Gregory Chanan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gregory Chanan updated SOLR-9200:
-
Attachment: SOLR-9200.patch

[~ichattopadhyaya] thanks so much!  I attached my current patch; I would very 
much appreciate a review.  My first couple of runs through the tests failed, 
although I haven't determined if those are just existing flaky tests or not.

I'll write up some notes shortly.

> Add Delegation Token Support to Solr
> 
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
>  Issue Type: New Feature
>  Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Attachments: SOLR-9200.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop 
> authentication filter.  Hadoop also has support for an authentication filter 
> that supports delegation tokens, which allow authenticated users the ability 
> to grab/renew/delete a token that can be used to bypass the normal 
> authentication path for a time.  This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access 
> to the user's kerberos credentials.  Instead, the job runner can grab a 
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can 
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more 
> privileged user can request a delegation token that can be passed to the less 
> privileged user.
> Note to self:
> In 
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
>  I made the following comment which I need to investigate further, since I 
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin 
> moving forward (I understand this is more a generic auth question than 
> kerberos specific). For example, in the latest version of the filter we are 
> using at Cloudera, we play around with the ServletContext in order to pass 
> information around 
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
>  Is there any way we can get the actual ServletContext in a plugin?{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9268) Support adding/updating backup repository configurations via API

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366427#comment-15366427
 ] 

Noble Paul edited comment on SOLR-9268 at 7/7/16 5:15 PM:
--

bq.Can you give a brief description of the intended design?

[~hgadre] The patch was to refactor out the mechanism used by security plugins 
into a base class so that all future plugins can reuse the code

The framework enables you to write  any {{ConfigEditablePlugin}} (i.e any 
plugin that supports editing its JSON configuration through an API). 


was (Author: noble.paul):
bq.Can you give a brief description of the intended design?

[~hgadre] The patch was to refactor out the mechanism used by security plugins 
into a base class so that all future plugins can reuse the code

The framework enables you to write  any {{ConfigEditablePlugin}} (i.e any 
plugin that supports editing it's json configuration through an API). 

> Support adding/updating backup repository configurations via API
> 
>
> Key: SOLR-9268
> URL: https://issues.apache.org/jira/browse/SOLR-9268
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
> Attachments: SOLR-9268.patch
>
>
> Currently users need to manually modify solr.xml in Zookeeper to update the 
> configuration parameters (and restart Solr cluster). This is not quite user 
> friendly. We should provide an API to update this configuration. (This came 
> up during the discussions in SOLR-9242).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9268) Support adding/updating backup repository configurations via API

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366427#comment-15366427
 ] 

Noble Paul commented on SOLR-9268:
--

bq.Can you give a brief description of the intended design?

[~hgadre] The patch was to refactor out the mechanism used by security plugins 
into a base class so that all future plugins can reuse the code

The framework enables you to write  any {{ConfigEditablePlugin}} (i.e any 
plugin that supports editing it's json configuration through an API). 

> Support adding/updating backup repository configurations via API
> 
>
> Key: SOLR-9268
> URL: https://issues.apache.org/jira/browse/SOLR-9268
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
> Attachments: SOLR-9268.patch
>
>
> Currently users need to manually modify solr.xml in Zookeeper to update the 
> configuration parameters (and restart Solr cluster). This is not quite user 
> friendly. We should provide an API to update this configuration. (This came 
> up during the discussions in SOLR-9242).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366417#comment-15366417
 ] 

Noble Paul commented on SOLR-7280:
--

Thanks [~markrmil...@gmail.com] for looking into this. The {{coreLoadThreads}} 
is something that's ignored in cloud mode today.  Should the user not have a 
way to limit the no:of threads used to load cores in cloud mode. If there are 
thousands of cores, the cluster ends up unstable because of this. Normally, 
users will have a few cores per node and it is no big deal to keep the thread 
pool to be unbounded. The power users must have a way to tune this. 

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366400#comment-15366400
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
Renamed to ComparisonBoolFunction


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #49: SOLR-9279 Adds comparison function queries

[softwaredoug]

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
Renamed to ComparisonBoolFunction


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_92) - Build # 17179 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17179/
Java: 32bit/jdk1.8.0_92 -client -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 10506 lines...]
[javac] Compiling 674 source files to 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/classes/test
[javac] 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:684: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/build.xml:59: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build.xml:233: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/common-build.xml:530: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:795: 
The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:807: 
The following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/common-build.xml:1955: 
Compile failed; see the compiler error output for details.

Total time: 28 minutes 32 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 251 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/251/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 10513 lines...]
[javac] Compiling 675 source files to 
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build/solr-core/classes/test
[javac] 
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:740: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:684: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/build.xml:59: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/build.xml:233: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/solr/common-build.xml:530:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:795:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:807:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-6.x-Solaris/lucene/common-build.xml:1955:
 Compile failed; see the compiler error output for details.

Total time: 19 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366396#comment-15366396
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
Renaming is a good idea. I can push that up. ComparisonFunction?

On Thu, Jul 7, 2016 at 12:13 PM David Smiley 
wrote:

> I think after addressing the comments I just added, it's probably good to
> go. I still don't love the name, especially since it extends BoolFunction
> it ought to now end with Function.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #49: SOLR-9279 Adds comparison function queries

[softwaredoug]

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
Renaming is a good idea. I can push that up. ComparisonFunction?

On Thu, Jul 7, 2016 at 12:13 PM David Smiley 
wrote:

> I think after addressing the comments I just added, it's probably good to
> go. I still don't love the name, especially since it extends BoolFunction
> it ought to now end with Function.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> ,
> or mute the thread
> 

> .
>



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8285) Ensure the /export handler works with NULL field values.

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366394#comment-15366394
 ] 

Erick Erickson commented on SOLR-8285:
--

Does https://issues.apache.org/jira/browse/SOLR-9166 address your question?

> Ensure the /export handler works with NULL field values.
> 
>
> Key: SOLR-8285
> URL: https://issues.apache.org/jira/browse/SOLR-8285
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Joel Bernstein
> Fix For: 6.0
>
> Attachments: SOLR-8285.patch, SOLR-8285.patch, SOLR-8285.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366385#comment-15366385
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit e91863b8e1b09c31e7b0c0e828b594ec9c022547 in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e91863b ]

SOLR-9236: Fix import.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7355) Leverage MultiTermAwareComponent in query parsers

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366384#comment-15366384
 ] 

Adrien Grand commented on LUCENE-7355:
--

I'll fix the docs to not bo specific to #normalize. I agree using 
attributeFactory() in tokenStream() has a large scope and probably deserves its 
own issue...

bq. BTW nice work on this issue; it's nice to see AnalyzingQueryParser go away 
and the lowercase options get removed.

Thanks!

> Leverage MultiTermAwareComponent in query parsers
> -
>
> Key: LUCENE-7355
> URL: https://issues.apache.org/jira/browse/LUCENE-7355
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, 
> LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch
>
>
> MultiTermAwareComponent is designed to make it possible to do the right thing 
> in query parsers when in comes to analysis of multi-term queries. However, 
> since query parsers just take an analyzer and since analyzers do not 
> propagate the information about what to do for multi-term analysis, query 
> parsers cannot do the right thing out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366381#comment-15366381
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit c58841558354c9b27fe502cad60907a62645bf3b in lucene-solr's branch 
refs/heads/master from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c588415 ]

SOLR-9236: Fix import.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7368) Remove queryNorm

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-7368:
-
Attachment: LUCENE-7368.patch

Here is a patch. getValueForNormalization and normalize are gone. Instead, 
boost are passed to Weights and stats at construction time. It helps have more 
final fields and simplifies BoostQuery. The boost is now a simple 
multiplicative factor, it can't get normalized away by the query norm like it 
used to happen with ClassicSimilarity.

> Remove queryNorm
> 
>
> Key: LUCENE-7368
> URL: https://issues.apache.org/jira/browse/LUCENE-7368
> Project: Lucene - Core
>  Issue Type: Sub-task
>Reporter: Adrien Grand
>Assignee: Adrien Grand
> Fix For: master (7.0)
>
> Attachments: LUCENE-7368.patch
>
>
> Splitting LUCENE-7347 into smaller tasks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 701 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/701/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 10450 lines...]
[javac] Compiling 674 source files to 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/classes/test
[javac] 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:740: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:684: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/build.xml:59: The 
following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build.xml:233: 
The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/common-build.xml:530:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:795:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:807:
 The following error occurred while executing this line:
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/lucene/common-build.xml:1955:
 Compile failed; see the compiler error output for details.

Total time: 20 minutes 52 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9268) Support adding/updating backup repository configurations via API

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366363#comment-15366363
 ] 

Varun Thacker commented on SOLR-9268:
-

For cluster level features I am of the opinion that we should have specific 
endpoints per feature.

For example basic auth has APIs to manage roles / users etc on a dedicated 
endpoint ( /solr/admin/authentication  )
https://cwiki.apache.org/confluence/display/solr/Basic+Authentication+Plugin

Similarly we should have an endpoint for backup repositories ( 
solr/admin/backup_repository ) where we can define the repositories and modify 
them dynamically. 

If we have one endpoint I feel it gets messy. Imagine if security / backup 
repository both could be configured via one endpoint ( 
/solr/admin/cluster_property ) , each with it's own json payload 



> Support adding/updating backup repository configurations via API
> 
>
> Key: SOLR-9268
> URL: https://issues.apache.org/jira/browse/SOLR-9268
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hrishikesh Gadre
> Attachments: SOLR-9268.patch
>
>
> Currently users need to manually modify solr.xml in Zookeeper to update the 
> configuration parameters (and restart Solr cluster). This is not quite user 
> friendly. We should provide an API to update this configuration. (This came 
> up during the discussions in SOLR-9242).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-MacOSX (64bit/jdk1.8.0) - Build # 261 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-MacOSX/261/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 10509 lines...]
[javac] Compiling 675 source files to 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build/solr-core/classes/test
[javac] 
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:740: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:684: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/build.xml:59: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/build.xml:233: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/solr/common-build.xml:530: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:795: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:807: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-6.x-MacOSX/lucene/common-build.xml:1955: 
Compile failed; see the compiler error output for details.

Total time: 24 minutes 9 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-6.x-Linux (32bit/jdk-9-ea+125) - Build # 1083 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1083/
Java: 32bit/jdk-9-ea+125 -server -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 10579 lines...]
[javac] Compiling 675 source files to 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/classes/test
[javac] 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:684: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:59: The following error 
occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:233: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:530: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:795: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:807: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:1955: 
Compile failed; see the compiler error output for details.

Total time: 33 minutes 1 second
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #49: SOLR-9279 Adds comparison function queries

[dsmiley]

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
I think after addressing the comments I just added, it's probably good to 
go.  I still don't love the name, especially since it extends BoolFunction it 
ought to now end with Function.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366326#comment-15366326
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
I think after addressing the comments I just added, it's probably good to 
go.  I still don't love the name, especially since it extends BoolFunction it 
ought to now end with Function.


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366318#comment-15366318
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69935281
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
+  public abstract boolean compare(double lhs, double rhs);
+
+  // Uniquely identify the operation (ie "gt", "lt" "gte", etc)
+  public String name() {
+return this.name;
+  }
+
+  // string comparison? Probably should be a seperate function
+  // public abstract boolean compareString(String lhs, String rhs);
+
+  public FunctionValues getValues(Map context, LeafReaderContext 
readerContext) throws IOException {
+final FunctionValues lhsVal = this.lhs.getValues(context, 
readerContext);
+final FunctionValues rhsVal = this.rhs.getValues(context, 
readerContext);
+final String compLabel = this.name();
+
+return new BoolDocValues(this) {
+  @Override
+  public boolean boolVal(int doc) {
+return compare(lhsVal.floatVal(doc), rhsVal.floatVal(doc));
--- End diff --

Can you instead pass lhsVal & rhsVal to let the compare() function choose 
if it wants to call floatVal, doubleVal, or longVal or whatever?


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling

[GitHub] lucene-solr pull request #49: SOLR-9279 Adds comparison function queries

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69935281
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
+  public abstract boolean compare(double lhs, double rhs);
+
+  // Uniquely identify the operation (ie "gt", "lt" "gte", etc)
+  public String name() {
+return this.name;
+  }
+
+  // string comparison? Probably should be a seperate function
+  // public abstract boolean compareString(String lhs, String rhs);
+
+  public FunctionValues getValues(Map context, LeafReaderContext 
readerContext) throws IOException {
+final FunctionValues lhsVal = this.lhs.getValues(context, 
readerContext);
+final FunctionValues rhsVal = this.rhs.getValues(context, 
readerContext);
+final String compLabel = this.name();
+
+return new BoolDocValues(this) {
+  @Override
+  public boolean boolVal(int doc) {
+return compare(lhsVal.floatVal(doc), rhsVal.floatVal(doc));
--- End diff --

Can you instead pass lhsVal & rhsVal to let the compare() function choose 
if it wants to call floatVal, doubleVal, or longVal or whatever?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9290) TCP-connections in CLOSE_WAIT spikes during heavy indexing when SSL is enabled

[Anshum Gupta (JIRA)]

Anshum Gupta created SOLR-9290:
--

 Summary: TCP-connections in CLOSE_WAIT spikes during heavy 
indexing when SSL is enabled
 Key: SOLR-9290
 URL: https://issues.apache.org/jira/browse/SOLR-9290
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 5.5.2, 5.5.1
Reporter: Anshum Gupta
Priority: Critical


Heavy indexing on Solr with SSL leads to a lot of connections in CLOSE_WAIT 
state. 

At my workplace, we have seen this issue only with 5.5.1 and could not 
reproduce it with 5.4.1 but from my conversation with Shalin, he knows of users 
with 5.3.1 running into this issue too. 

Here's an excerpt from the email [~shaie] sent to the mailing list  (about what 
we see:

{quote}
1) It consistently reproduces on 5.5.1, but *does not* reproduce on 5.4.1
2) It does not reproduce when SSL is disabled
3) Restarting the Solr process (sometimes both need to be restarted), the
count drops to 0, but if indexing continues, they climb up again

When it does happen, Solr seems stuck. The leader cannot talk to the
replica, or vice versa, the replica is usually put in DOWN state and
there's no way to fix it besides restarting the JVM.
{quote}

Here's the mail thread: 
http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201607.mbox/%3c46cc66220a8143dc903fa34e79205...@vp-exc01.dips.local%3E

Creating this issue so we could track this and have more people comment on what 
they see. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366316#comment-15366316
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69934986
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
--- End diff --

Use javadoc method comments, not //


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #49: SOLR-9279 Adds comparison function queries

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69934986
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
--- End diff --

Use javadoc method comments, not //


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366314#comment-15366314
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69934693
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
+  public abstract boolean compare(double lhs, double rhs);
+
+  // Uniquely identify the operation (ie "gt", "lt" "gte", etc)
+  public String name() {
+return this.name;
+  }
+
+  // string comparison? Probably should be a seperate function
+  // public abstract boolean compareString(String lhs, String rhs);
+
+  public FunctionValues getValues(Map context, LeafReaderContext 
readerContext) throws IOException {
+final FunctionValues lhsVal = this.lhs.getValues(context, 
readerContext);
+final FunctionValues rhsVal = this.rhs.getValues(context, 
readerContext);
+final String compLabel = this.name();
+
+return new BoolDocValues(this) {
+  @Override
+  public boolean boolVal(int doc) {
+return compare(lhsVal.floatVal(doc), rhsVal.floatVal(doc));
+  }
+
+  @Override
+  public String toString(int doc) {
+return compLabel + "(" + lhsVal.toString(doc) + "," + 
rhsVal.toString(doc) + ")";
+  }
+};
+  }
+
+  @Override
+  public boolean equals(Object o) {
+if (this.getClass() != o.getClass()) return false;
+if (!(o instanceof ComparisonValueSource)) return false;
--- End diff --

This line is not needed; the classes must be equal in the previous line.


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if 

[GitHub] lucene-solr pull request #49: SOLR-9279 Adds comparison function queries

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/49#discussion_r69934693
  
--- Diff: 
lucene/queries/src/java/org/apache/lucene/queries/function/valuesource/ComparisonValueSource.java
 ---
@@ -0,0 +1,104 @@
+package org.apache.lucene.queries.function.valuesource;
+
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import java.io.IOException;
+import java.util.Map;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.queries.function.ValueSource;
+import org.apache.lucene.queries.function.FunctionValues;
+import org.apache.lucene.queries.function.docvalues.BoolDocValues;
+import org.apache.lucene.search.IndexSearcher;
+
+
+/**
+ * Base class for comparison operators used within if statements
+ * To Solr's if function query a 0 is considered "false", all other values 
are "true"
+ */
+public abstract class ComparisonValueSource extends BoolFunction {
+
+  private final ValueSource lhs;
+  private final ValueSource rhs;
+  private final String name;
+
+  public ComparisonValueSource(ValueSource lhs, ValueSource rhs, String 
name) {
+this.lhs = lhs;
+this.rhs = rhs;
+this.name = name;
+  }
+
+  // Perform the comparison, returning true or false
+  public abstract boolean compare(double lhs, double rhs);
+
+  // Uniquely identify the operation (ie "gt", "lt" "gte", etc)
+  public String name() {
+return this.name;
+  }
+
+  // string comparison? Probably should be a seperate function
+  // public abstract boolean compareString(String lhs, String rhs);
+
+  public FunctionValues getValues(Map context, LeafReaderContext 
readerContext) throws IOException {
+final FunctionValues lhsVal = this.lhs.getValues(context, 
readerContext);
+final FunctionValues rhsVal = this.rhs.getValues(context, 
readerContext);
+final String compLabel = this.name();
+
+return new BoolDocValues(this) {
+  @Override
+  public boolean boolVal(int doc) {
+return compare(lhsVal.floatVal(doc), rhsVal.floatVal(doc));
+  }
+
+  @Override
+  public String toString(int doc) {
+return compLabel + "(" + lhsVal.toString(doc) + "," + 
rhsVal.toString(doc) + ")";
+  }
+};
+  }
+
+  @Override
+  public boolean equals(Object o) {
+if (this.getClass() != o.getClass()) return false;
+if (!(o instanceof ComparisonValueSource)) return false;
--- End diff --

This line is not needed; the classes must be equal in the previous line.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3393 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3393/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

All tests passed

Build Log:
[...truncated 10449 lines...]
[javac] Compiling 674 source files to 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/classes/test
[javac] 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:740: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:684: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/build.xml:59: The following 
error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build.xml:233: The 
following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/common-build.xml:530: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:795: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:807: 
The following error occurred while executing this line:
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/common-build.xml:1955:
 Compile failed; see the compiler error output for details.

Total time: 25 minutes 50 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_92) - Build # 17178 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17178/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 891 lines...]
   [junit4] JVM J2: stdout was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/temp/junit4-J2-20160707_153124_153.sysout
   [junit4] >>> JVM J2 emitted unexpected output (verbatim) 
   [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0xf6ee5829, pid=9940, tid=0xe65ffb40
   [junit4] #
   [junit4] # JRE version: Java(TM) SE Runtime Environment (8.0_92-b14) (build 
1.8.0_92-b14)
   [junit4] # Java VM: Java HotSpot(TM) Server VM (25.92-b14 mixed mode 
linux-x86 )
   [junit4] # Problematic frame:
   [junit4] # V  [libjvm.so+0x73a829]  
ObjArrayKlass::oop_oop_iterate_range_nv(oopDesc*, ParScanWithBarrierClosure*, 
int, int)+0x89
   [junit4] #
   [junit4] # Failed to write core dump. Core dumps have been disabled. To 
enable core dumping, try "ulimit -c unlimited" before starting Java again
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # 
/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J2/hs_err_pid9940.log
   [junit4] #
   [junit4] # If you would like to submit a bug report, please visit:
   [junit4] #   http://bugreport.java.com/bugreport/crash.jsp
   [junit4] #
   [junit4] <<< JVM J2: EOF 

[...truncated 732 lines...]
   [junit4] ERROR: JVM J2 ended with an exception, command line: 
/home/jenkins/tools/java/32bit/jdk1.8.0_92/jre/bin/java -server 
-XX:+UseConcMarkSweepGC -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/home/jenkins/workspace/Lucene-Solr-master-Linux/heapdumps -ea 
-esa -Dtests.prefix=tests -Dtests.seed=41B237501452D837 -Xmx512M -Dtests.iters= 
-Dtests.verbose=false -Dtests.infostream=false -Dtests.codec=random 
-Dtests.postingsformat=random -Dtests.docvaluesformat=random 
-Dtests.locale=random -Dtests.timezone=random -Dtests.directory=random 
-Dtests.linedocsfile=europarl.lines.txt.gz -Dtests.luceneMatchVersion=7.0.0 
-Dtests.cleanthreads=perMethod 
-Djava.util.logging.config.file=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/junit4/logging.properties
 -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false 
-Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=3 -DtempDir=./temp 
-Djava.io.tmpdir=./temp 
-Djunit4.tempDir=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/temp
 -Dcommon.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene 
-Dclover.db.dir=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/clover/db
 
-Djava.security.policy=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/tools/junit4/tests.policy
 -Dtests.LUCENE_VERSION=7.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 
-Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory 
-Djava.awt.headless=true -Djdk.map.althashing.threshold=0 
-Djunit4.childvm.cwd=/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/build/core/test/J2
 -Djunit4.childvm.id=2 -Djunit4.childvm.count=3 -Dtests.leaveTemporary=false 
-Dtests.filterstacks=true -Dtests.disableHdfs=true 
-Djava.security.manager=org.apache.lucene.util.TestSecurityManager 
-Dfile.encoding=UTF-8 -classpath 

[jira] [Comment Edited] (SOLR-9252) Feature selection and logistic regression on text

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366283#comment-15366283
 ] 

Joel Bernstein edited comment on SOLR-9252 at 7/7/16 3:40 PM:
--

[~caomanhdat], I have the patch applied and have begun the review.

I've started with the FeaturesSelectionStream and IGainTermsQParserPlugin. I'll 
need more time and some collaboration to review the math. But I can say now 
that the mechanics of feature selection look very good. The use of the 
streaming framework and analytics query is really nice.

The one thing that we'll want to do is put some thought into how the features 
can be stored and retrieved.

Currently it looks like there is a tuple for each term/score pair. I think this 
works well using the update() function to send the tuples to another collection 
for storage. A few minor things to consider:

1) Should we use a field type post fix (term_s , score_f) to ensure that fields 
are indexed properly in another collection.

2) We'll need to add some kind of feature set ID so the feature set can be 
retrieved later. Each tuple will then be tagged with the feature set ID. 
Possibly adding a *featureSet* parameter to the stream makes sense for this.
 
3) We can also add a unique ID which will be used for the unique ID for each 
tuple in the index. We could concat the term with the feature set ID to make 
the unique ID.








was (Author: joel.bernstein):
[~caomanhdat], I have the patch applied and have begun the review.

I've started with the FeaturesSelectionStream and IGainTermsQParserPlugin. I'll 
need more time and some collaboration to review the math. But I can say now 
that the mechanics of feature selection look very good. The use of the 
streaming framework and analytics query is really nice.

The one thing that we'll want to do is put some thought into how the features 
can be stored and retrieved.

Currently it looks like there is a tuple for each term/score pair. I think this 
works well using the update() function to send the tuples to another collection 
for storage. A few minor things to consider:

1) Should we use a field type post fix (term_s , score_f) to ensure that fields 
are indexed properly in another collection.

2) We'll need to add some kind of feature set ID so the feature set can be 
retrieved later. Each tuple will then be tagged with the feature set ID. 
Possibly adding a *featureSet* parameter to the stream makes sense for this.
 
3) We can also add a unique ID which will be used for the unique ID for each 
tuple in the index. We could concat the the term with the feature set ID to 
make the unique ID.







> Feature selection and logistic regression on text
> -
>
> Key: SOLR-9252
> URL: https://issues.apache.org/jira/browse/SOLR-9252
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: SOLR-9252.patch, SOLR-9252.patch, enron1.zip
>
>
> SOLR-9186 come up with a challenges that for each iterative we have to 
> rebuild the tf-idf vector for each documents. It is costly computation if we 
> represent doc by a lot of terms. Features selection can help reducing the 
> computation.
> Due to its computational efficiency and simple interpretation, information 
> gain is one of the most popular feature selection methods. It is used to 
> measure the dependence between features and labels and calculates the 
> information gain between the i-th feature and the class labels 
> (http://www.jiliang.xyz/publication/feature_selection_for_classification.pdf).
> I confirmed that by running logistics regressions on enron mail dataset (in 
> which each email is represented by top 100 terms that have highest 
> information gain) and got the accuracy by 92% and precision by 82%.
> This ticket will create two new streaming expression. Both of them use the 
> same *parallel iterative framework* as SOLR-8492.
> {code}
> featuresSelection(collection1, q="*:*",  field="tv_text", outcome="out_i", 
> positiveLabel=1, numTerms=100)
> {code}
> featuresSelection will emit top terms that have highest information gain 
> scores. It can be combined with new tlogit stream.
> {code}
> tlogit(collection1, q="*:*",
>  featuresSelection(collection1, 
>   q="*:*",  
>   field="tv_text", 
>   outcome="out_i", 
>   positiveLabel=1, 
>   numTerms=100),
>  field="tv_text",
>  outcome="out_i",
>  maxIterations=100)
> {code}
> In the iteration n, the text logistics regression will emit nth model, and 
> compute the 

[jira] [Commented] (SOLR-9252) Feature selection and logistic regression on text

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366283#comment-15366283
 ] 

Joel Bernstein commented on SOLR-9252:
--

[~caomanhdat], I have the patch applied and have begun the review.

I've started with the FeaturesSelectionStream and IGainTermsQParserPlugin. I'll 
need more time and some collaboration to review the math. But I can say now 
that the mechanics of feature selection look very good. The use of the 
streaming framework and analytics query is really nice.

The one thing that we'll want to do is put some thought into how the features 
can be stored and retrieved.

Currently it looks like there is a tuple for each term/score pair. I think this 
works well using the update() function to send the tuples to another collection 
for storage. A few minor things to consider:

1) Should we use a field type post fix (term_s , score_f) to ensure that fields 
are indexed properly in another collection.

2) We'll need to add some kind of feature set ID so the feature set can be 
retrieved later. Each tuple will then be tagged with the feature set ID. 
Possibly adding a *featureSet* parameter to the stream makes sense for this.
 
3) We can also add a unique ID which will be used for the unique ID for each 
tuple in the index. We could concat the the term with the feature set ID to 
make the unique ID.







> Feature selection and logistic regression on text
> -
>
> Key: SOLR-9252
> URL: https://issues.apache.org/jira/browse/SOLR-9252
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: SOLR-9252.patch, SOLR-9252.patch, enron1.zip
>
>
> SOLR-9186 come up with a challenges that for each iterative we have to 
> rebuild the tf-idf vector for each documents. It is costly computation if we 
> represent doc by a lot of terms. Features selection can help reducing the 
> computation.
> Due to its computational efficiency and simple interpretation, information 
> gain is one of the most popular feature selection methods. It is used to 
> measure the dependence between features and labels and calculates the 
> information gain between the i-th feature and the class labels 
> (http://www.jiliang.xyz/publication/feature_selection_for_classification.pdf).
> I confirmed that by running logistics regressions on enron mail dataset (in 
> which each email is represented by top 100 terms that have highest 
> information gain) and got the accuracy by 92% and precision by 82%.
> This ticket will create two new streaming expression. Both of them use the 
> same *parallel iterative framework* as SOLR-8492.
> {code}
> featuresSelection(collection1, q="*:*",  field="tv_text", outcome="out_i", 
> positiveLabel=1, numTerms=100)
> {code}
> featuresSelection will emit top terms that have highest information gain 
> scores. It can be combined with new tlogit stream.
> {code}
> tlogit(collection1, q="*:*",
>  featuresSelection(collection1, 
>   q="*:*",  
>   field="tv_text", 
>   outcome="out_i", 
>   positiveLabel=1, 
>   numTerms=100),
>  field="tv_text",
>  outcome="out_i",
>  maxIterations=100)
> {code}
> In the iteration n, the text logistics regression will emit nth model, and 
> compute the error of (n-1)th model. Because the error will be wrong if we 
> compute the error dynamically in each iteration. 
> In each iteration tlogit will change learning rate based on error of previous 
> iteration. It will increase the learning rate by 5% if error is going down 
> and It will decrease the learning rate by 50% if error is going up.
> This will support use cases such as building models for spam detection, 
> sentiment analysis and threat detection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Windows (32bit/jdk1.8.0_92) - Build # 302 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/302/
Java: 32bit/jdk1.8.0_92 -server -XX:+UseG1GC

All tests passed

Build Log:
[...truncated 10569 lines...]
[javac] Compiling 675 source files to 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build\solr-core\classes\test
[javac] 
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\core\src\test\org\apache\solr\cloud\SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\build.xml:740: The following 
error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\build.xml:684: The following 
error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\build.xml:59: The following 
error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\build.xml:233: The 
following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\solr\common-build.xml:530: 
The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\lucene\common-build.xml:795: 
The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\lucene\common-build.xml:807: 
The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-6.x-Windows\lucene\common-build.xml:1955:
 Compile failed; see the compiler error output for details.

Total time: 21 minutes 28 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7888) Make Lucene's AnalyzingInfixSuggester.lookup() method that takes a BooleanQuery filter parameter available in Solr

[Rajesh Kapur (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366267#comment-15366267
 ] 

Rajesh Kapur commented on SOLR-7888:


Hi,

Can we pass multiple fields to be filtered out in ContextField parameter? If 
yes, please give me example. 

> Make Lucene's AnalyzingInfixSuggester.lookup() method that takes a 
> BooleanQuery filter parameter available in Solr
> --
>
> Key: SOLR-7888
> URL: https://issues.apache.org/jira/browse/SOLR-7888
> Project: Solr
>  Issue Type: New Feature
>  Components: Suggester
>Affects Versions: 5.2.1
>Reporter: Arcadius Ahouansou
>Assignee: Jan Høydahl
> Fix For: 5.4, 6.0
>
> Attachments: SOLR-7888-7963.patch, SOLR-7888.patch, SOLR-7888.patch
>
>
>  LUCENE-6464 has introduced a very flexible lookup method that takes as 
> parameter a BooleanQuery that is used for filtering results.
> This ticket is to expose that method to Solr.
> This would allow user to do:
> {code}
> /suggest?suggest=true=true=term=contexts:tennis
> /suggest?suggest=true=true=term=contexts:golf
>  AND contexts:football
> {code}
> etc
> Given that the context filtering in currently only implemented by the 
> {code}AnalyzingInfixSuggester{code} and by the 
> {code}BlendedInfixSuggester{code}, this initial implementation will support 
> only these 2 lookup implementations.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9163) Confusing solrconfig.xml in the downloaded solr*.zip

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366258#comment-15366258
 ] 

Yonik Seeley commented on SOLR-9163:


Is everyone OK with committing this as a first (possibly only) step?
I don't have further time to work on this right now.

> Confusing solrconfig.xml in the downloaded solr*.zip
> 
>
> Key: SOLR-9163
> URL: https://issues.apache.org/jira/browse/SOLR-9163
> Project: Solr
>  Issue Type: Bug
>Reporter: Sachin Goyal
> Attachments: SOLR-9163.patch, SOLR-9163.patch
>
>
> Here are the solrconfig.xml when I download and unzip solr:
> {code}
> find . -name solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/db/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/mail/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/rss/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/solr/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/tika/conf/solrconfig.xml
> ./solr-5.5.1/example/files/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/basic_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
> {code}
> Most likely, the ones I want to use are in server/solr/configsets, I assume.
> But then which ones among those three?
> Searching online does not provide much detailed information.
> And diff-ing among them yields even more confusing results.
> Example: When I diff basic_configs/conf/solrconfig.xml with 
> data_driven_schema_configs/conf/solrconfig.xml, I am not sure why the latter 
> has these extra constrcuts?
> # solr.LimitTokenCountFilterFactory and all the comments around it.
> # deletionPolicy class="solr.SolrDeletionPolicy"
> # Commented out infoStream file="INFOSTREAM.txt"
> # Extra comments for "Update Related Event Listeners"
> # indexReaderFactory
> # And so for lots of other constructs and comments.
> The point is that it is difficult to find out exactly what extra features in 
> the latter are making it data-driven. Hence it is difficult to know what 
> features I am losing by not taking the data-driven-schema.
> It would be good to sync the above 3 files together (each file should have 
> same comments and differ only in the configuration which makes them 
> different). Also, some good documentation should be put online about them 
> otherwise it is very confusing for non-committers and vanilla-users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366260#comment-15366260
 ] 

Mark Miller commented on SOLR-9236:
---

Whoops. Removed commenting out the nightly annotation, but bringing back the 
missing import didn't make the commit. 

> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366253#comment-15366253
 ] 

Steve Rowe commented on SOLR-9236:
--

[~markrmil...@gmail.com], test compilation is broken on master & branch_6x, I 
think because of this issue - here's the branch_6x error I get:

{noformat}
common.compile-test:
[mkdir] Created dir: 
/Users/sarowe/git/lucene-solr-2/solr/build/solr-core/classes/test
[javac] Compiling 675 source files to 
/Users/sarowe/git/lucene-solr-2/solr/build/solr-core/classes/test
[javac] 
/Users/sarowe/git/lucene-solr-2/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error
{noformat}

> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-6.x-Linux (64bit/jdk-9-ea+125) - Build # 1082 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1082/
Java: 64bit/jdk-9-ea+125 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 10579 lines...]
[javac] Compiling 675 source files to 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build/solr-core/classes/test
[javac] 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/core/src/test/org/apache/solr/cloud/SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:740: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:684: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:59: The following error 
occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:233: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:530: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:795: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:807: The 
following error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/common-build.xml:1955: 
Compile failed; see the compiler error output for details.

Total time: 23 minutes 53 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_92) - Build # 5965 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/5965/
Java: 32bit/jdk1.8.0_92 -client -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 10515 lines...]
[javac] Compiling 674 source files to 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\classes\test
[javac] 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\core\src\test\org\apache\solr\cloud\SharedFSAutoReplicaFailoverTest.java:65:
 error: cannot find symbol
[javac] @Nightly
[javac]  ^
[javac]   symbol: class Nightly
[javac] Note: Some input files use or override a deprecated API.
[javac] Note: Recompile with -Xlint:deprecation for details.
[javac] Note: Some input files use unchecked or unsafe operations.
[javac] Note: Recompile with -Xlint:unchecked for details.
[javac] 1 error

BUILD FAILED
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:740: The 
following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:684: The 
following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\build.xml:59: The 
following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build.xml:233: The 
following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\common-build.xml:530:
 The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:795:
 The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:807:
 The following error occurred while executing this line:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\common-build.xml:1955:
 Compile failed; see the compiler error output for details.

Total time: 26 minutes 57 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9163) Confusing solrconfig.xml in the downloaded solr*.zip

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366232#comment-15366232
 ] 

David Smiley commented on SOLR-9163:


[~noble.paul] would the "config overlay" thing work here?  I mistakenly 
suggested params.json I think.

> Confusing solrconfig.xml in the downloaded solr*.zip
> 
>
> Key: SOLR-9163
> URL: https://issues.apache.org/jira/browse/SOLR-9163
> Project: Solr
>  Issue Type: Bug
>Reporter: Sachin Goyal
> Attachments: SOLR-9163.patch, SOLR-9163.patch
>
>
> Here are the solrconfig.xml when I download and unzip solr:
> {code}
> find . -name solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/db/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/mail/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/rss/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/solr/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/tika/conf/solrconfig.xml
> ./solr-5.5.1/example/files/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/basic_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
> {code}
> Most likely, the ones I want to use are in server/solr/configsets, I assume.
> But then which ones among those three?
> Searching online does not provide much detailed information.
> And diff-ing among them yields even more confusing results.
> Example: When I diff basic_configs/conf/solrconfig.xml with 
> data_driven_schema_configs/conf/solrconfig.xml, I am not sure why the latter 
> has these extra constrcuts?
> # solr.LimitTokenCountFilterFactory and all the comments around it.
> # deletionPolicy class="solr.SolrDeletionPolicy"
> # Commented out infoStream file="INFOSTREAM.txt"
> # Extra comments for "Update Related Event Listeners"
> # indexReaderFactory
> # And so for lots of other constructs and comments.
> The point is that it is difficult to find out exactly what extra features in 
> the latter are making it data-driven. Hence it is difficult to know what 
> features I am losing by not taking the data-driven-schema.
> It would be good to sync the above 3 files together (each file should have 
> same comments and differ only in the configuration which makes them 
> different). Also, some good documentation should be put online about them 
> otherwise it is very confusing for non-committers and vanilla-users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9163) Confusing solrconfig.xml in the downloaded solr*.zip

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366201#comment-15366201
 ] 

Yonik Seeley commented on SOLR-9163:


bq. Yonik is this just the first step in addition to some mechanism for us to 
ensure they stay in sync? 

Maybe..., I don't know what the next step is, and I think manual syncing is the 
first step regardless.
Hopefully at least committers will see that they are 99% the same and help keep 
them that way.
I was just looking into params.json but they didn't quite work the way I 
thought.  I had to specify params.json in the request to get it to work.

> Confusing solrconfig.xml in the downloaded solr*.zip
> 
>
> Key: SOLR-9163
> URL: https://issues.apache.org/jira/browse/SOLR-9163
> Project: Solr
>  Issue Type: Bug
>Reporter: Sachin Goyal
> Attachments: SOLR-9163.patch, SOLR-9163.patch
>
>
> Here are the solrconfig.xml when I download and unzip solr:
> {code}
> find . -name solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/db/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/mail/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/rss/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/solr/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/tika/conf/solrconfig.xml
> ./solr-5.5.1/example/files/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/basic_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
> {code}
> Most likely, the ones I want to use are in server/solr/configsets, I assume.
> But then which ones among those three?
> Searching online does not provide much detailed information.
> And diff-ing among them yields even more confusing results.
> Example: When I diff basic_configs/conf/solrconfig.xml with 
> data_driven_schema_configs/conf/solrconfig.xml, I am not sure why the latter 
> has these extra constrcuts?
> # solr.LimitTokenCountFilterFactory and all the comments around it.
> # deletionPolicy class="solr.SolrDeletionPolicy"
> # Commented out infoStream file="INFOSTREAM.txt"
> # Extra comments for "Update Related Event Listeners"
> # indexReaderFactory
> # And so for lots of other constructs and comments.
> The point is that it is difficult to find out exactly what extra features in 
> the latter are making it data-driven. Hence it is difficult to know what 
> features I am losing by not taking the data-driven-schema.
> It would be good to sync the above 3 files together (each file should have 
> same comments and differ only in the configuration which makes them 
> different). Also, some good documentation should be put online about them 
> otherwise it is very confusing for non-committers and vanilla-users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #49: SOLR-9279 Adds comparison function queries

[softwaredoug]

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
(note the last few commits based on your comments)

On unifying somewhat with MultiBoolValues, or creating a BiBoolValues, I 
went down that path a bit David and it seems to complicate things. A couple of 
notes:

- The bool values functions takes as input other boolfunctions, whereas the 
comparison value source takes in scalar values.  You can see this in how or, 
and, xor work: they loop over several boolean value sources and perform and, 
or, xor etc. We just need to pluck out two scalar values and compare them
- The name `func` seems descriptive of this general behavior, whereas 
`compare` is more descriptive of the operation being perfomed by the comparison 
value source

I think the comparison functions are more readable now as they are, but I'd 
be curious to get your thoughts.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9279) Add greater than, less than, etc in Solr function queries

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366187#comment-15366187
 ] 

ASF GitHub Bot commented on SOLR-9279:
--

Github user softwaredoug commented on the issue:

https://github.com/apache/lucene-solr/pull/49
  
(note the last few commits based on your comments)

On unifying somewhat with MultiBoolValues, or creating a BiBoolValues, I 
went down that path a bit David and it seems to complicate things. A couple of 
notes:

- The bool values functions takes as input other boolfunctions, whereas the 
comparison value source takes in scalar values.  You can see this in how or, 
and, xor work: they loop over several boolean value sources and perform and, 
or, xor etc. We just need to pluck out two scalar values and compare them
- The name `func` seems descriptive of this general behavior, whereas 
`compare` is more descriptive of the operation being perfomed by the comparison 
value source

I think the comparison functions are more readable now as they are, but I'd 
be curious to get your thoughts.


> Add greater than, less than, etc in Solr function queries
> -
>
> Key: SOLR-9279
> URL: https://issues.apache.org/jira/browse/SOLR-9279
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Reporter: Doug Turnbull
> Fix For: master (7.0)
>
>
> If you use the "if" function query, you'll often expect to be able to use 
> greater than/less than functions. For example, you might want to boost books 
> written in the past 7 years. Unfortunately, there's no "greater than" 
> function query that will return non-zero when the lhs > rhs. Instead to get 
> this, you need to create really awkward function queries like I do here 
> (http://opensourceconnections.com/blog/2014/11/26/stepwise-date-boosting-in-solr/):
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> The pull request attached to this Jira adds the following function queries
> (https://github.com/apache/lucene-solr/pull/49)
> -gt(lhs, rhs) (returns 1 if lhs > rhs, 0 otherwise)
> -lt(lhs, rhs) (returns 1 if lhs < rhs, 0 otherwise)
> -gte
> -lte
> -eq
> So instead of 
> if(min(0,sub(ms(mydatefield),sub(ms(NOW),315569259747))),0.8,1)
> one could now write
> if(lt(ms(mydatefield),315569259747,0.8,1)
> (if mydatefield < 315569259747 then 0.8 else 1)
> A bit more readable and less puzzling



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9166) Export handler returns zero for numeric fields that are not in the original doc

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9166:
-
Summary: Export handler returns zero for numeric fields that are not in the 
original doc  (was: Export handler returns zero for fields numeric fields that 
are not in the original doc)

> Export handler returns zero for numeric fields that are not in the original 
> doc
> ---
>
> Key: SOLR-9166
> URL: https://issues.apache.org/jira/browse/SOLR-9166
> Project: Solr
>  Issue Type: Bug
>Reporter: Erick Erickson
>Assignee: Erick Erickson
> Attachments: SOLR-9166.patch
>
>
> From the dev list discussion:
> My original post.
> Zero is different from not
> existing. And let's claim that I want to process a stream and, say,
> facet on in integer field over the result set. There's no way on the
> client side to distinguish between a document that has a zero in the
> field and one that didn't have the field in the first place so I'll
> over-count the zero bucket.
> From Dennis Gove:
> Is this true for non-numeric fields as well? I agree that this seems like a 
> very bad thing.
> I can't imagine that a fix would cause a problem with Streaming Expressions, 
> ParallelSQL, or other given that the /select handler is not returning 0 for 
> these missing fields (the /select handler is the default handler for the 
> Streaming API so if nulls were a problem I imagine we'd have already seen 
> it). 
> That said, within Streaming Expressions there is a select(...) function which 
> supports a replace(...) operation which allows you to replace one value (or 
> null) with some other value. If a 0 were necessary one could use a 
> select(...) to replace null with 0 using an expression like this 
>select(, replace(fieldA, null, withValue=0)). 
> The end result of that would be that the field fieldA would never have a null 
> value and for all tuples where a null value existed it would be replaced with 
> 0.
> Details on the select function can be found at 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61330338#StreamingExpressions-select.
> And to answer Denis' question, null gets returned for string DocValues fields.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9163) Confusing solrconfig.xml in the downloaded solr*.zip

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366171#comment-15366171
 ] 

David Smiley commented on SOLR-9163:


Yonik is this just the first step in addition to some mechanism for us to 
ensure they stay in sync?  Without some mechanism, they will fall out of sync 
again.

> Confusing solrconfig.xml in the downloaded solr*.zip
> 
>
> Key: SOLR-9163
> URL: https://issues.apache.org/jira/browse/SOLR-9163
> Project: Solr
>  Issue Type: Bug
>Reporter: Sachin Goyal
> Attachments: SOLR-9163.patch, SOLR-9163.patch
>
>
> Here are the solrconfig.xml when I download and unzip solr:
> {code}
> find . -name solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/db/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/mail/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/rss/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/solr/conf/solrconfig.xml
> ./solr-5.5.1/example/example-DIH/solr/tika/conf/solrconfig.xml
> ./solr-5.5.1/example/files/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/basic_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/data_driven_schema_configs/conf/solrconfig.xml
> ./solr-5.5.1/server/solr/configsets/sample_techproducts_configs/conf/solrconfig.xml
> {code}
> Most likely, the ones I want to use are in server/solr/configsets, I assume.
> But then which ones among those three?
> Searching online does not provide much detailed information.
> And diff-ing among them yields even more confusing results.
> Example: When I diff basic_configs/conf/solrconfig.xml with 
> data_driven_schema_configs/conf/solrconfig.xml, I am not sure why the latter 
> has these extra constrcuts?
> # solr.LimitTokenCountFilterFactory and all the comments around it.
> # deletionPolicy class="solr.SolrDeletionPolicy"
> # Commented out infoStream file="INFOSTREAM.txt"
> # Extra comments for "Update Related Event Listeners"
> # indexReaderFactory
> # And so for lots of other constructs and comments.
> The point is that it is difficult to find out exactly what extra features in 
> the latter are making it data-driven. Hence it is difficult to know what 
> features I am losing by not taking the data-driven-schema.
> It would be good to sync the above 3 files together (each file should have 
> same comments and differ only in the configuration which makes them 
> different). Also, some good documentation should be put online about them 
> otherwise it is very confusing for non-committers and vanilla-users.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7280) Load cores in sorted order and tweak coreLoadThread counts to improve cluster stability on restarts

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366168#comment-15366168
 ] 

Mark Miller commented on SOLR-7280:
---

bq. // This assumes replicaCount is less than coreLoadThreadCount?

Why is that a question?

If you don't have enough threads to load all the cores in a shard, you have to 
wait for the leader vote timeout which is long.

That doesn't seem look good default behavior. How does this address that? I'm 
only mildly caught up on the related issue.

> Load cores in sorted order and tweak coreLoadThread counts to improve cluster 
> stability on restarts
> ---
>
> Key: SOLR-7280
> URL: https://issues.apache.org/jira/browse/SOLR-7280
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
> Fix For: 5.2, 6.0
>
> Attachments: SOLR-7280.patch
>
>
> In SOLR-7191, Damien mentioned that by loading solr cores in a sorted order 
> and tweaking some of the coreLoadThread counts, he was able to improve the 
> stability of a cluster with thousands of collections. We should explore some 
> of these changes and fold them into Solr.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7355) Leverage MultiTermAwareComponent in query parsers

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366159#comment-15366159
 ] 

David Smiley commented on LUCENE-7355:
--

I like the new Analyzer.attributeFactory() method but I don't like that it 
documents that it's for #normalize -- as if it should only be used for 
normalize.  Wouldn't it be useful for createComponents() too?  That would be a 
bigger change, however, since there are lots of times when a Tokenizer is 
created within the context of an Analyzer that would ideally be updated to call 
this method.  That seems like it deserves its own issue?  Or maybe for the time 
being we will accept that it's currently only used by normalize.  It would be 
nice to see CustomAnalyzer have a customizable AttributeFactory for TokenStream 
and to be returned by this proposed method.

bq. That said, maybe the method approach is better since borrowing the 
attribute factory from the regular analysis chain makes us close the token 
stream before it has been consumed, which some analysis chains might not like.

I think token streams should be tolerant of this or something in the TS chain 
is broken IMO.

RE Solr, I only mean if there is an  and 
thus the actual chain is opaque to Solr so it can't use it's normal means of 
determining the default multiTerm analysis chain.  This is a bit of a fringe 
issue any way since in my experience setting class= is rare.

BTW nice work on this issue; it's nice to see AnalyzingQueryParser go away and 
the lowercase options get removed.

> Leverage MultiTermAwareComponent in query parsers
> -
>
> Key: LUCENE-7355
> URL: https://issues.apache.org/jira/browse/LUCENE-7355
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch, 
> LUCENE-7355.patch, LUCENE-7355.patch, LUCENE-7355.patch
>
>
> MultiTermAwareComponent is designed to make it possible to do the right thing 
> in query parsers when in comes to analysis of multi-term queries. However, 
> since query parsers just take an analyzer and since analyzers do not 
> propagate the information about what to do for multi-term analysis, query 
> parsers cannot do the right thing out of the box.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-9252) Feature selection and logistic regression on text

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein reassigned SOLR-9252:


Assignee: Joel Bernstein

> Feature selection and logistic regression on text
> -
>
> Key: SOLR-9252
> URL: https://issues.apache.org/jira/browse/SOLR-9252
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Cao Manh Dat
>Assignee: Joel Bernstein
> Attachments: SOLR-9252.patch, SOLR-9252.patch, enron1.zip
>
>
> SOLR-9186 come up with a challenges that for each iterative we have to 
> rebuild the tf-idf vector for each documents. It is costly computation if we 
> represent doc by a lot of terms. Features selection can help reducing the 
> computation.
> Due to its computational efficiency and simple interpretation, information 
> gain is one of the most popular feature selection methods. It is used to 
> measure the dependence between features and labels and calculates the 
> information gain between the i-th feature and the class labels 
> (http://www.jiliang.xyz/publication/feature_selection_for_classification.pdf).
> I confirmed that by running logistics regressions on enron mail dataset (in 
> which each email is represented by top 100 terms that have highest 
> information gain) and got the accuracy by 92% and precision by 82%.
> This ticket will create two new streaming expression. Both of them use the 
> same *parallel iterative framework* as SOLR-8492.
> {code}
> featuresSelection(collection1, q="*:*",  field="tv_text", outcome="out_i", 
> positiveLabel=1, numTerms=100)
> {code}
> featuresSelection will emit top terms that have highest information gain 
> scores. It can be combined with new tlogit stream.
> {code}
> tlogit(collection1, q="*:*",
>  featuresSelection(collection1, 
>   q="*:*",  
>   field="tv_text", 
>   outcome="out_i", 
>   positiveLabel=1, 
>   numTerms=100),
>  field="tv_text",
>  outcome="out_i",
>  maxIterations=100)
> {code}
> In the iteration n, the text logistics regression will emit nth model, and 
> compute the error of (n-1)th model. Because the error will be wrong if we 
> compute the error dynamically in each iteration. 
> In each iteration tlogit will change learning rate based on error of previous 
> iteration. It will increase the learning rate by 5% if error is going down 
> and It will decrease the learning rate by 50% if error is going up.
> This will support use cases such as building models for spam detection, 
> sentiment analysis and threat detection. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366110#comment-15366110
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit dc763e383d4293f9dc235bcc63ae2fba582574ff in lucene-solr's branch 
refs/heads/branch_6x from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=dc763e3 ]

SOLR-9236: AutoAddReplicas will append an extra /tlog to the update log 
location on replica failover.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9236) AutoAddReplicas feature with one replica loses some documents not committed during failover

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15366107#comment-15366107
 ] 

ASF subversion and git services commented on SOLR-9236:
---

Commit 546093812c34610075ee130f7466eca1979cfbeb in lucene-solr's branch 
refs/heads/master from markrmiller
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5460938 ]

SOLR-9236: AutoAddReplicas will append an extra /tlog to the update log 
location on replica failover.


> AutoAddReplicas feature with one replica loses some documents not committed 
> during failover
> ---
>
> Key: SOLR-9236
> URL: https://issues.apache.org/jira/browse/SOLR-9236
> Project: Solr
>  Issue Type: Bug
>  Components: hdfs, SolrCloud
>Reporter: Eungsop Yoo
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-9236.patch, SOLR-9236.patch
>
>
> I need to index huge amount of logs, so I decide to use AutoAddReplica 
> feature with only one replica.
> When using AutoAddReplicas with one replica, some benefits are expected.
> - no redundant data files for replicas
> -- saving disk usage
> - best indexing performance 
> I expected that Solr fails over just like HBase.
> The feature worked almost as it was expected, except for some missing 
> documents during failover.
> I found two regions for the missing.
> 1. The leader replica does not replay any transaction logs. But when there is 
> only one replica, it should be the leader.
> So I made the leader replica replay the transaction logs when using 
> AutoAddReplicas with on replica.
> But the above fix did not resolve the problem.
> 2. As failover occurred, the transaction log directory had a deeper directory 
> depth. Just like this, tlog/tlog/tlog/...
> The transaction log could not be replayed, because the transaction log 
> directory was changed during failover. 
> So I made the transaction log directory not changed during failover.
> After these fixes, AutoAddReplicas with one replica fails over well without 
> losing any documents.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 700 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/700/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
Expected 2 of 3 replicas to be active but only found 1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica1","base_url":"http://127.0.0.1:61356","node_name":"127.0.0.1:61356_","state":"active","leader":"true"}];
 clusterState: 
DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/20)={   
"replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node1":{   "core":"c8n_1x3_lf_shard1_replica3",   
"base_url":"http://127.0.0.1:43159;,   "node_name":"127.0.0.1:43159_",  
 "state":"down"}, "core_node2":{   "state":"down",  
 "base_url":"http://127.0.0.1:47038;,   
"core":"c8n_1x3_lf_shard1_replica2",   "node_name":"127.0.0.1:47038_"}, 
"core_node3":{   "core":"c8n_1x3_lf_shard1_replica1",   
"base_url":"http://127.0.0.1:61356;,   "node_name":"127.0.0.1:61356_",  
 "state":"active",   "leader":"true",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false"}

Stack Trace:
java.lang.AssertionError: Expected 2 of 3 replicas to be active but only found 
1; 
[core_node3:{"core":"c8n_1x3_lf_shard1_replica1","base_url":"http://127.0.0.1:61356","node_name":"127.0.0.1:61356_","state":"active","leader":"true"}];
 clusterState: DocCollection(c8n_1x3_lf//collections/c8n_1x3_lf/state.json/20)={
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node1":{
  "core":"c8n_1x3_lf_shard1_replica3",
  "base_url":"http://127.0.0.1:43159;,
  "node_name":"127.0.0.1:43159_",
  "state":"down"},
"core_node2":{
  "state":"down",
  "base_url":"http://127.0.0.1:47038;,
  "core":"c8n_1x3_lf_shard1_replica2",
  "node_name":"127.0.0.1:47038_"},
"core_node3":{
  "core":"c8n_1x3_lf_shard1_replica1",
  "base_url":"http://127.0.0.1:61356;,
  "node_name":"127.0.0.1:61356_",
  "state":"active",
  "leader":"true",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false"}
at 
__randomizedtesting.SeedInfo.seed([BA0A3546EADC5FE3:325E0A9C4420321B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:170)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:57)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at