[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 3556 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/3556/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
at 
__randomizedtesting.SeedInfo.seed([840FCF0BA6890C5B:737C21536061A3BD]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1329)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+136) - Build # 17872 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17872/
Java: 64bit/jdk-9-ea+136 -XX:+UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamExpressionTest

Error Message:
10 threads leaked from SUITE scope at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest: 1) 
Thread[id=1750, 
name=TEST-StreamExpressionTest.testBasicTextLogitStream-seed#[2E384439FEB094F2]-SendThread(127.0.0.1:34524),
 state=TIMED_WAITING, group=TGRP-StreamExpressionTest] at 
java.lang.Thread.sleep(java.base@9-ea/Native Method) at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
 at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
2) Thread[id=1970, name=zkCallback-746-thread-2, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at 
jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:232)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:462)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:361)
 at 
java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1085)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1146)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:635)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)3) 
Thread[id=1763, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at 
java.lang.Thread.sleep(java.base@9-ea/Native Method) at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)4) 
Thread[id=1751, 
name=TEST-StreamExpressionTest.testBasicTextLogitStream-seed#[2E384439FEB094F2]-EventThread,
 state=WAITING, group=TGRP-StreamExpressionTest] at 
jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at 
java.util.concurrent.locks.LockSupport.park(java.base@9-ea/LockSupport.java:192)
 at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(java.base@9-ea/AbstractQueuedSynchronizer.java:2062)
 at 
java.util.concurrent.LinkedBlockingQueue.take(java.base@9-ea/LinkedBlockingQueue.java:442)
 at 
org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)5) 
Thread[id=1764, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at 
java.lang.Thread.sleep(java.base@9-ea/Native Method) at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)6) 
Thread[id=1752, name=zkCallback-746-thread-1, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at 
jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:232)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:462)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:361)
 at 
java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1085)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@9-ea/ThreadPoolExecutor.java:1146)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@9-ea/ThreadPoolExecutor.java:635)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)7) 
Thread[id=1762, name=TextLogitSolrStream-747-thread-2, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at 
jdk.internal.misc.Unsafe.park(java.base@9-ea/Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(java.base@9-ea/LockSupport.java:232)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(java.base@9-ea/SynchronousQueue.java:462)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(java.base@9-ea/SynchronousQueue.java:361)
 at 
java.util.concurrent.SynchronousQueue.poll(java.base@9-ea/SynchronousQueue.java:937)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(java.base@9-ea/ThreadPoolExecutor.java:1085)
 at 

[JENKINS] Lucene-Solr-6.x-Windows (64bit/jdk1.8.0_102) - Build # 467 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/467/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
Timeout occured while waiting response from server at: http://127.0.0.1:63500/w_

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:63500/w_
at 
__randomizedtesting.SeedInfo.seed([30FC3DF44A6A0F5C:B8A8022EE49662A4]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:603)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:405)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:357)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1599)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1553)
at 
org.apache.solr.cloud.CustomCollectionTest.testCustomCollectionsAPI(CustomCollectionTest.java:285)
at 
org.apache.solr.cloud.CustomCollectionTest.test(CustomCollectionTest.java:99)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 

[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_102) - Build # 1773 - Still Failing!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1773/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 37973 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.LogManager 
[Use slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:126)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Logger [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:126)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Appender [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:130)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.LogManager 
[Use slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:131)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Logger [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:131)
[forbidden-apis] Scanned 3313 (and 2152 related) class file(s) for forbidden 
API invocations (in 2.27s), 5 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:763: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:366: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:514: Check 
for forbidden API calls failed, see log.

Total time: 61 minutes 43 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_102) - Build # 17871 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17871/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testUpdateStream

Error Message:
Error from server at http://127.0.0.1:33004/solr: collection already exists: 
destinationCollection

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:33004/solr: collection already exists: 
destinationCollection
at 
__randomizedtesting.SeedInfo.seed([DDAD1328CF42E6EA:15EAD752C64A1BBA]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:415)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:367)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testUpdateStream(StreamExpressionTest.java:2700)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6130 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6130/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
ObjectTracker found 10 object(s) that were not released!!! 
[MockDirectoryWrapper, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
TransactionLog, MockDirectoryWrapper, TransactionLog, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not 
released!!! [MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
MockDirectoryWrapper, TransactionLog, MockDirectoryWrapper, TransactionLog, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper]
at __randomizedtesting.SeedInfo.seed([29104CDF0DCB0491]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
at sun.reflect.GeneratedMethodAccessor18.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
Could not remove the following files (in the order of attempts):
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_29104CDF0DCB0491-001\tempDir-001\node1\testschemaapi_shard1_replica1\data\tlog\tlog.000:
 java.nio.file.FileSystemException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_29104CDF0DCB0491-001\tempDir-001\node1\testschemaapi_shard1_replica1\data\tlog\tlog.000:
 The process cannot access the file because it is being used by another 
process. 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_29104CDF0DCB0491-001\tempDir-001\node1\testschemaapi_shard1_replica1\data\tlog:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_29104CDF0DCB0491-001\tempDir-001\node1\testschemaapi_shard1_replica1\data\tlog

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J0\temp\solr.schema.TestManagedSchemaAPI_29104CDF0DCB0491-001\tempDir-001\node1\testschemaapi_shard1_replica1\data:
 java.nio.file.DirectoryNotEmptyException: 

[jira] [Commented] (SOLR-6871) Need a process for updating & maintaining the new quickstart tutorial (and any other tutorials added to the website)

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511717#comment-15511717
 ] 

Steve Rowe commented on SOLR-6871:
--

I committed the updated quickstart tutorial in both places.  I'll leave the 
issue open for a little while in case anybody wants to do more here.

> Need a process for updating & maintaining the new quickstart tutorial (and 
> any other tutorials added to the website)
> 
>
> Key: SOLR-6871
> URL: https://issues.apache.org/jira/browse/SOLR-6871
> Project: Solr
>  Issue Type: Task
>Reporter: Hoss Man
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-6871.patch
>
>
> Prior to SOLR-6058 the /solr/tutorial.html link on the website contained only 
> a simple landing page that then linked people to the "versioned" tutorial for 
> the most recent release -- or more specificly: the most recent release*s* 
> (plural) when we were releasing off of multiple branches (ie: links to both 
> the 4.0.0 tutorial, as well as the 3.6.3 tutorial when 4.0 came out)
> The old tutorial content lived along side the solr code, and was 
> automatically branched, tagged & released along with Solr.  When committing 
> any changes to Solr code (or post.jar code, or the sample data, or the sample 
> configs, etc..) you could also commit changes to the tutorial at th same time 
> and be confident that it was clear what version of solr that tutorial went 
> along with.
> As part of SOLR-6058, it seems that there was a concensus to move to a 
> keeping "tutorial" content on the website, where it can be integrated 
> directly in with other site content/navigation, and use the same look and 
> feel.
> I have no objection to this in principle -- but as a result of this choice, 
> there are outstanding issues regarding how devs should go about maintaining 
> this doc as changes are made to solr & the solr examples used in the tutorial.
> We need a clear process for where/how to edit the tutorial(s) as new versions 
> of solr come out and cahnges are made that mandate corisponding hanges to the 
> tutorial.  this process _should_ also account for things like having multiple 
> versions of the tutorial live at one time (ie: at some point in the future, 
> we'll certainly need to host the "5.13" tutorial if that's the current 
> "stable" release, but we'll also want to host the tutorial for "6.0-BETA" so 
> that people can try it out)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_102) - Build # 1772 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1772/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseConcMarkSweepGC

All tests passed

Build Log:
[...truncated 37990 lines...]
-check-forbidden-all:
[forbidden-apis] Reading bundled API signatures: jdk-unsafe-1.8
[forbidden-apis] Reading bundled API signatures: jdk-deprecated-1.8
[forbidden-apis] Reading bundled API signatures: jdk-non-portable
[forbidden-apis] Reading bundled API signatures: jdk-reflection
[forbidden-apis] Reading bundled API signatures: commons-io-unsafe-2.5
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/base.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/servlet-api.txt
[forbidden-apis] Reading API signatures: 
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/lucene/tools/forbiddenApis/solr.txt
[forbidden-apis] Loading classes to check...
[forbidden-apis] Scanning classes for violations...
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.LogManager 
[Use slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:126)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Logger [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:126)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Appender [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:130)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.LogManager 
[Use slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:131)
[forbidden-apis] Forbidden class/interface use: org.apache.log4j.Logger [Use 
slf4j classes instead]
[forbidden-apis]   in org.apache.solr.servlet.SolrDispatchFilter 
(SolrDispatchFilter.java:131)
[forbidden-apis] Scanned 3313 (and 2152 related) class file(s) for forbidden 
API invocations (in 2.71s), 5 error(s).

BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:763: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/build.xml:117: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/build.xml:347: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-6.x-Linux/solr/common-build.xml:514: Check 
for forbidden API calls failed, see log.

Total time: 69 minutes 13 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any



-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6871) Need a process for updating & maintaining the new quickstart tutorial (and any other tutorials added to the website)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511680#comment-15511680
 ] 

ASF subversion and git services commented on SOLR-6871:
---

Commit 1761836 from [~steve_rowe] in branch 'cms/trunk'
[ https://svn.apache.org/r1761836 ]

SOLR-6871: Updated the quickstart tutorial to cover the 6.2.0 release

> Need a process for updating & maintaining the new quickstart tutorial (and 
> any other tutorials added to the website)
> 
>
> Key: SOLR-6871
> URL: https://issues.apache.org/jira/browse/SOLR-6871
> Project: Solr
>  Issue Type: Task
>Reporter: Hoss Man
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-6871.patch
>
>
> Prior to SOLR-6058 the /solr/tutorial.html link on the website contained only 
> a simple landing page that then linked people to the "versioned" tutorial for 
> the most recent release -- or more specificly: the most recent release*s* 
> (plural) when we were releasing off of multiple branches (ie: links to both 
> the 4.0.0 tutorial, as well as the 3.6.3 tutorial when 4.0 came out)
> The old tutorial content lived along side the solr code, and was 
> automatically branched, tagged & released along with Solr.  When committing 
> any changes to Solr code (or post.jar code, or the sample data, or the sample 
> configs, etc..) you could also commit changes to the tutorial at th same time 
> and be confident that it was clear what version of solr that tutorial went 
> along with.
> As part of SOLR-6058, it seems that there was a concensus to move to a 
> keeping "tutorial" content on the website, where it can be integrated 
> directly in with other site content/navigation, and use the same look and 
> feel.
> I have no objection to this in principle -- but as a result of this choice, 
> there are outstanding issues regarding how devs should go about maintaining 
> this doc as changes are made to solr & the solr examples used in the tutorial.
> We need a clear process for where/how to edit the tutorial(s) as new versions 
> of solr come out and cahnges are made that mandate corisponding hanges to the 
> tutorial.  this process _should_ also account for things like having multiple 
> versions of the tutorial live at one time (ie: at some point in the future, 
> we'll certainly need to host the "5.13" tutorial if that's the current 
> "stable" release, but we'll also want to host the tutorial for "6.0-BETA" so 
> that people can try it out)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9534) Support quiet/verbose bin/solr options for changing log level

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511633#comment-15511633
 ] 

Jan Høydahl commented on SOLR-9534:
---

(/) Tested on Windows 10
I plan to commit this tomorrow, please voice any concerns.
PS: These start options are of course just an alternative to the existing, i.e. 
selecting another log4j.properties or editing the default one.

> Support quiet/verbose bin/solr options for changing log level
> -
>
> Key: SOLR-9534
> URL: https://issues.apache.org/jira/browse/SOLR-9534
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: scripts and tools
>Affects Versions: 6.2
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9534.patch
>
>
> Spinoff from SOLR-6677
> Let's make it much easier to "turn on debug" by supporting a {{bin/solr start 
> -V}} verbose option, and likewise a {{bin/solr start -q}} for quiet operation.
> These would simply be convenience options for changing the RootLogger from 
> level INFO to DEBUG or WARN respectively. This can be done programmatically 
> in log4j at startup. 
> Could be we need to add some more package specific defaults in 
> log4j.properties to get the right mix of logs



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6871) Need a process for updating & maintaining the new quickstart tutorial (and any other tutorials added to the website)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511594#comment-15511594
 ] 

ASF subversion and git services commented on SOLR-6871:
---

Commit 53981795fd73e85aae1892c3c72344af7c57083a in lucene-solr's branch 
refs/heads/master from [~steve_rowe]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5398179 ]

SOLR-6871: Updated the quickstart tutorial to cover the 6.2.0 release, and 
added ant target "generate-website-quickstart" to convert the bundled version 
of the tutorial into one suitable for the website.


> Need a process for updating & maintaining the new quickstart tutorial (and 
> any other tutorials added to the website)
> 
>
> Key: SOLR-6871
> URL: https://issues.apache.org/jira/browse/SOLR-6871
> Project: Solr
>  Issue Type: Task
>Reporter: Hoss Man
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-6871.patch
>
>
> Prior to SOLR-6058 the /solr/tutorial.html link on the website contained only 
> a simple landing page that then linked people to the "versioned" tutorial for 
> the most recent release -- or more specificly: the most recent release*s* 
> (plural) when we were releasing off of multiple branches (ie: links to both 
> the 4.0.0 tutorial, as well as the 3.6.3 tutorial when 4.0 came out)
> The old tutorial content lived along side the solr code, and was 
> automatically branched, tagged & released along with Solr.  When committing 
> any changes to Solr code (or post.jar code, or the sample data, or the sample 
> configs, etc..) you could also commit changes to the tutorial at th same time 
> and be confident that it was clear what version of solr that tutorial went 
> along with.
> As part of SOLR-6058, it seems that there was a concensus to move to a 
> keeping "tutorial" content on the website, where it can be integrated 
> directly in with other site content/navigation, and use the same look and 
> feel.
> I have no objection to this in principle -- but as a result of this choice, 
> there are outstanding issues regarding how devs should go about maintaining 
> this doc as changes are made to solr & the solr examples used in the tutorial.
> We need a clear process for where/how to edit the tutorial(s) as new versions 
> of solr come out and cahnges are made that mandate corisponding hanges to the 
> tutorial.  this process _should_ also account for things like having multiple 
> versions of the tutorial live at one time (ie: at some point in the future, 
> we'll certainly need to host the "5.13" tutorial if that's the current 
> "stable" release, but we'll also want to host the tutorial for "6.0-BETA" so 
> that people can try it out)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6871) Need a process for updating & maintaining the new quickstart tutorial (and any other tutorials added to the website)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511593#comment-15511593
 ] 

ASF subversion and git services commented on SOLR-6871:
---

Commit 8984e73660427b4be37f5eae177fbf8697e2e0c0 in lucene-solr's branch 
refs/heads/branch_6x from [~steve_rowe]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8984e73 ]

SOLR-6871: Updated the quickstart tutorial to cover the 6.2.0 release, and 
added ant target "generate-website-quickstart" to convert the bundled version 
of the tutorial into one suitable for the website.


> Need a process for updating & maintaining the new quickstart tutorial (and 
> any other tutorials added to the website)
> 
>
> Key: SOLR-6871
> URL: https://issues.apache.org/jira/browse/SOLR-6871
> Project: Solr
>  Issue Type: Task
>Reporter: Hoss Man
>Assignee: Steve Rowe
>Priority: Minor
> Attachments: SOLR-6871.patch
>
>
> Prior to SOLR-6058 the /solr/tutorial.html link on the website contained only 
> a simple landing page that then linked people to the "versioned" tutorial for 
> the most recent release -- or more specificly: the most recent release*s* 
> (plural) when we were releasing off of multiple branches (ie: links to both 
> the 4.0.0 tutorial, as well as the 3.6.3 tutorial when 4.0 came out)
> The old tutorial content lived along side the solr code, and was 
> automatically branched, tagged & released along with Solr.  When committing 
> any changes to Solr code (or post.jar code, or the sample data, or the sample 
> configs, etc..) you could also commit changes to the tutorial at th same time 
> and be confident that it was clear what version of solr that tutorial went 
> along with.
> As part of SOLR-6058, it seems that there was a concensus to move to a 
> keeping "tutorial" content on the website, where it can be integrated 
> directly in with other site content/navigation, and use the same look and 
> feel.
> I have no objection to this in principle -- but as a result of this choice, 
> there are outstanding issues regarding how devs should go about maintaining 
> this doc as changes are made to solr & the solr examples used in the tutorial.
> We need a clear process for where/how to edit the tutorial(s) as new versions 
> of solr come out and cahnges are made that mandate corisponding hanges to the 
> tutorial.  this process _should_ also account for things like having multiple 
> versions of the tutorial live at one time (ie: at some point in the future, 
> we'll certainly need to host the "5.13" tutorial if that's the current 
> "stable" release, but we'll also want to host the tutorial for "6.0-BETA" so 
> that people can try it out)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8495) Schemaless mode cannot index large text fields

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511574#comment-15511574
 ] 

Steve Rowe commented on SOLR-8495:
--

I looked at [~caomanhdat]'s patch, and I think there's way more machinery there 
than we need to address the problem.  A couple things I noticed:

* ChunkTokenizer splits values at a maximum token length (rather than 
truncating), but I can't think of a good use for that behavior.
* ParseLongStringFieldUpdateProcessorFactory extends 
NumericFieldUpdateProcessorFactory, which doesn't make sense, since there's no 
parsing going on, and LongStringField isn't numeric. 
* ParseLongStringFieldUpdateProcessor.mutateValue() uses 
String.getBytes(Charset.defaultCharset()) to determine a value's length, but 
Lucene will use UTF-8 to string terms, so UTF-8 should be used when testing 
value lengths. 
* I don't think we need new tokenizers or processors or field types here.

I agree with [~hossman] that his SOLR-9526 approach is the way to go (including 
his TruncateFieldUpdateProcessorFactory idea mentioned above, to address the 
problem described on this issue - his suggested "1" limit neatly avoids 
worrying about encoded length issues, since each char can take up at most 3 
UTF-8 encoded bytes, and 3*1 is less than the 32,766 byte 
IndexWriter.MAX_TERM_LENGTH.)

{quote}
bq. Autodetect space-separated text above a (customizable? maybe 256 bytes or 
so by default?) threshold as tokenized text rather than as StrField.
I'm leary of this an approach like this, because it would be extremely trappy 
depending on the order docs were indexed
{quote}

I agree, hoss'ss SOLR-9526 approach will index everything as text_general but 
then add "string" fieldtype copies for values that aren't "too long".

> Schemaless mode cannot index large text fields
> --
>
> Key: SOLR-8495
> URL: https://issues.apache.org/jira/browse/SOLR-8495
> Project: Solr
>  Issue Type: Bug
>  Components: Data-driven Schema, Schema and Analysis
>Affects Versions: 4.10.4, 5.3.1, 5.4
>Reporter: Shalin Shekhar Mangar
>  Labels: difficulty-easy, impact-medium
> Fix For: 5.5, 6.0
>
> Attachments: SOLR-8495.patch
>
>
> The schemaless mode by default indexes all string fields into an indexed 
> StrField which is limited to 32KB text. Anything larger than that leads to an 
> exception during analysis.
> {code}
> Caused by: java.lang.IllegalArgumentException: Document contains at least one 
> immense term in field="text" (whose UTF8 encoding is longer than the max 
> length 32766)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_102) - Build # 17870 - Still Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17870/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseG1GC

4 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
ObjectTracker found 6 object(s) that were not released!!! [SolrCore, SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 6 object(s) that were not 
released!!! [SolrCore, SolrCore, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper]
at __randomizedtesting.SeedInfo.seed([FD8F3262AB2FE149]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
at sun.reflect.GeneratedMethodAccessor19.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testFeaturesSelectionStream

Error Message:
Error from server at https://127.0.0.1:45111/solr: collection already exists: 
destinationCollection

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:45111/solr: collection already exists: 
destinationCollection
at 
__randomizedtesting.SeedInfo.seed([6CE914EA2BB99FB0:76B98FD4BDFEFD6]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:415)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:367)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testFeaturesSelectionStream(StreamExpressionTest.java:3231)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 

[jira] [Resolved] (SOLR-8186) Solr start scripts -- only log to console when running in foreground

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-8186.
---
Resolution: Fixed

> Solr start scripts -- only log to console when running in foreground
> 
>
> Key: SOLR-8186
> URL: https://issues.apache.org/jira/browse/SOLR-8186
> Project: Solr
>  Issue Type: Improvement
>  Components: scripts and tools
>Affects Versions: 5.3.1
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8186.patch, SOLR-8186.patch
>
>
> Currently the log4j.properties file logs to the console, and the start 
> scripts capture console output to a logfile that never rotates.  This can 
> fill up the disk, and when the logfile is removed, the user might be alarmed 
> by the way their memory statistics behave -- the "cached" memory might have a 
> sudden and very large drop, making it appear to a novice that the huge 
> logfile was hogging their memory.
> The logfile created by log4j is rotated when it gets big enough, so that 
> logfile is unlikely to fill up the disk.
> I propose that we copy the current log4j.properties file to something like 
> log4j-foreground.properties, remove CONSOLE logging in the log4j.properties 
> file, and have the start script use the alternate config file when running in 
> the foreground.  This way users will see the logging output when running in 
> the foreground, but it will be absent when running normally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9548) solr.log should start with informative welcome message

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511532#comment-15511532
 ] 

Shawn Heisey commented on SOLR-9548:


No objections.  The line is fairly long, though.  A thought for consideration:  
Would it make sense to output multiple lines that are each shorter, to reduce 
the risk of the lines wrapping on an 80-column text console?

> solr.log should start with informative welcome message
> --
>
> Key: SOLR-9548
> URL: https://issues.apache.org/jira/browse/SOLR-9548
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9548.patch
>
>
> When starting Solr, the first log line should be more informative, such as
> {code}
> Welcome to Apache Solr™ version 7.0.0, running in standalone mode on port 
> 8983 from folder /Users/janhoy/git/lucene-solr/solr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8186) Solr start scripts -- only log to console when running in foreground

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511535#comment-15511535
 ] 

ASF subversion and git services commented on SOLR-8186:
---

Commit dfb8aff7394057e132c3b3ca0ef107f280cca8df in lucene-solr's branch 
refs/heads/branch_6x from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=dfb8aff ]

SOLR-8186: Solr start scripts, only log to console when running in foreground


> Solr start scripts -- only log to console when running in foreground
> 
>
> Key: SOLR-8186
> URL: https://issues.apache.org/jira/browse/SOLR-8186
> Project: Solr
>  Issue Type: Improvement
>  Components: scripts and tools
>Affects Versions: 5.3.1
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8186.patch, SOLR-8186.patch
>
>
> Currently the log4j.properties file logs to the console, and the start 
> scripts capture console output to a logfile that never rotates.  This can 
> fill up the disk, and when the logfile is removed, the user might be alarmed 
> by the way their memory statistics behave -- the "cached" memory might have a 
> sudden and very large drop, making it appear to a novice that the huge 
> logfile was hogging their memory.
> The logfile created by log4j is rotated when it gets big enough, so that 
> logfile is unlikely to fill up the disk.
> I propose that we copy the current log4j.properties file to something like 
> log4j-foreground.properties, remove CONSOLE logging in the log4j.properties 
> file, and have the start script use the alternate config file when running in 
> the foreground.  This way users will see the logging output when running in 
> the foreground, but it will be absent when running normally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5563) Tone down some SolrCloud logging

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511529#comment-15511529
 ] 

Jan Høydahl commented on SOLR-5563:
---

This looks very good.
I think we just need to fix the logging one bite at a time and commit stuff as 
we go. This is a huge step wrt readability of Cloud logging! Then we can tackle 
shutdown cases next perhaps, could be a bit more tricky to detect normal cases 
from the irregular ones perhaps?

> Tone down some SolrCloud logging
> 
>
> Key: SOLR-5563
> URL: https://issues.apache.org/jira/browse/SOLR-5563
> Project: Solr
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Attachments: SOLR-5563.patch, SOLR-5563.patch
>
>
> We output a *lot* of logging data from various bits of SolrCloud, a lot of 
> which is basically implementation detail that isn't particularly helpful.  
> Here's a patch to move a bunch of stuff to #debug level, rather than #info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8186) Solr start scripts -- only log to console when running in foreground

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15504932#comment-15504932
 ] 

Jan Høydahl edited comment on SOLR-8186 at 9/21/16 11:20 PM:
-

Attached patch with the following:

* New option {{-Dsolr.log.muteconsole}} which is passed when starting in 
foreground mode ({{-f}}). This will programatically disable the {{CONSOLE}} 
logger (in SolrDispatchFilter.init), causing the {{solr-8983-console.log}} to 
only contain stdout/stderr logs (except for the first few lines before the 
logger is disabled).
* -Removed some excess Jetty logging by setting default level for 
{{org.eclipse.jetty=WARN}} and {{org.eclipse.jetty.server=INFO}}-
* Removed annoying log line {{o.e.j.s.SecurityHandler ... has uncovered http 
methods for path: /}} by extending web.xml
* Removed annoying log line {{o.a.s.c.CoreContainer Couldn't add files from 
/opt/solr/server/solr/lib to classpath:}} when libPath is the hardcoded {{lib}}
* Now printing full date also for CONSOLE log

I decided to do the dynamic disabling of CONSOLE logger instead of having 
multiple {{log4j.properties}} files floating around, meaning that the muting 
will work also for custom logger configs, as long as the console logger is 
named {{CONSOLE}}. This is more flexible.


was (Author: janhoy):
Attached patch with the following:

* New option {{-Dsolr.log.muteconsole}} which is passed when starting in 
foreground mode ({{-f}}). This will programatically disable the {{CONSOLE}} 
logger (in SolrDispatchFilter.init), causing the {{solr-8983-console.log}} to 
only contain stdout/stderr logs (except for the first few lines before the 
logger is disabled).
* Removed some excess Jetty logging by setting default level for 
{{org.eclipse.jetty=WARN}} and {{org.eclipse.jetty.server=INFO}}
* Removed annoying log line {{o.e.j.s.SecurityHandler ... has uncovered http 
methods for path: /}} by extending web.xml
* Removed annoying log line {{o.a.s.c.CoreContainer Couldn't add files from 
/opt/solr/server/solr/lib to classpath:}} when libPath is the hardcoded {{lib}}
* Now printing full date also for CONSOLE log

I decided to do the dynamic disabling of CONSOLE logger instead of having 
multiple {{log4j.properties}} files floating around, meaning that the muting 
will work also for custom logger configs, as long as the console logger is 
named {{CONSOLE}}. This is more flexible.

> Solr start scripts -- only log to console when running in foreground
> 
>
> Key: SOLR-8186
> URL: https://issues.apache.org/jira/browse/SOLR-8186
> Project: Solr
>  Issue Type: Improvement
>  Components: scripts and tools
>Affects Versions: 5.3.1
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8186.patch, SOLR-8186.patch
>
>
> Currently the log4j.properties file logs to the console, and the start 
> scripts capture console output to a logfile that never rotates.  This can 
> fill up the disk, and when the logfile is removed, the user might be alarmed 
> by the way their memory statistics behave -- the "cached" memory might have a 
> sudden and very large drop, making it appear to a novice that the huge 
> logfile was hogging their memory.
> The logfile created by log4j is rotated when it gets big enough, so that 
> logfile is unlikely to fill up the disk.
> I propose that we copy the current log4j.properties file to something like 
> log4j-foreground.properties, remove CONSOLE logging in the log4j.properties 
> file, and have the start script use the alternate config file when running in 
> the foreground.  This way users will see the logging output when running in 
> the foreground, but it will be absent when running normally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8186) Solr start scripts -- only log to console when running in foreground

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511501#comment-15511501
 ] 

ASF subversion and git services commented on SOLR-8186:
---

Commit 2ef34e9e24ac2c0bd72de54c3edda9ba3aa43728 in lucene-solr's branch 
refs/heads/master from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2ef34e9 ]

SOLR-8186: Solr start scripts, only log to console when running in foreground


> Solr start scripts -- only log to console when running in foreground
> 
>
> Key: SOLR-8186
> URL: https://issues.apache.org/jira/browse/SOLR-8186
> Project: Solr
>  Issue Type: Improvement
>  Components: scripts and tools
>Affects Versions: 5.3.1
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8186.patch, SOLR-8186.patch
>
>
> Currently the log4j.properties file logs to the console, and the start 
> scripts capture console output to a logfile that never rotates.  This can 
> fill up the disk, and when the logfile is removed, the user might be alarmed 
> by the way their memory statistics behave -- the "cached" memory might have a 
> sudden and very large drop, making it appear to a novice that the huge 
> logfile was hogging their memory.
> The logfile created by log4j is rotated when it gets big enough, so that 
> logfile is unlikely to fill up the disk.
> I propose that we copy the current log4j.properties file to something like 
> log4j-foreground.properties, remove CONSOLE logging in the log4j.properties 
> file, and have the start script use the alternate config file when running in 
> the foreground.  This way users will see the logging output when running in 
> the foreground, but it will be absent when running normally.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-9548) solr.log should start with informative welcome message

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl reassigned SOLR-9548:
-

Assignee: Jan Høydahl

> solr.log should start with informative welcome message
> --
>
> Key: SOLR-9548
> URL: https://issues.apache.org/jira/browse/SOLR-9548
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9548.patch
>
>
> When starting Solr, the first log line should be more informative, such as
> {code}
> Welcome to Apache Solr™ version 7.0.0, running in standalone mode on port 
> 8983 from folder /Users/janhoy/git/lucene-solr/solr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9548) solr.log should start with informative welcome message

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-9548:
--
Fix Version/s: master (7.0)
   6.3

> solr.log should start with informative welcome message
> --
>
> Key: SOLR-9548
> URL: https://issues.apache.org/jira/browse/SOLR-9548
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9548.patch
>
>
> When starting Solr, the first log line should be more informative, such as
> {code}
> Welcome to Apache Solr™ version 7.0.0, running in standalone mode on port 
> 8983 from folder /Users/janhoy/git/lucene-solr/solr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9548) solr.log should start with informative welcome message

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-9548:
--
Attachment: SOLR-9548.patch

Simple patch. I'm assuming {{jetty.port}} and {{solr.install.dir}} are always 
set, and that the presence of either sysprop {{zkHost}} or {{zkRun}} is a 
certain way of detecting Cloud mode.

After reducing more log noise this will be the 2nd line in a fresh {{solr.log}} 
file. Since this log line says almost exactly the same as the echo line from 
the start script when starting in foreground, this patch removes that echo.

Startup will then look like
{noformat}
% bin/solr start -f
162  INFO  (main) [   ] o.e.j.s.Server jetty-9.3.8.v20160314
500  INFO  (main) [   ] o.a.s.s.SolrDispatchFilter Welcome to Apache Solr™ 
version 7.0.0, starting in standalone mode on port 8983 from 
/Users/janhoy/git/lucene-solr/solr
524  INFO  (main) [   ] o.a.s.c.SolrResourceLoader using system property 
solr.solr.home: /Users/janhoy/git/lucene-solr/solr/server/solr
...
{noformat}

Any objections to this approach?

> solr.log should start with informative welcome message
> --
>
> Key: SOLR-9548
> URL: https://issues.apache.org/jira/browse/SOLR-9548
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>  Labels: logging
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9548.patch
>
>
> When starting Solr, the first log line should be more informative, such as
> {code}
> Welcome to Apache Solr™ version 7.0.0, running in standalone mode on port 
> 8983 from folder /Users/janhoy/git/lucene-solr/solr
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9548) solr.log should start with informative welcome message

[JIRA]

Jan Høydahl created SOLR-9548:
-

 Summary: solr.log should start with informative welcome message
 Key: SOLR-9548
 URL: https://issues.apache.org/jira/browse/SOLR-9548
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Server
Reporter: Jan Høydahl


When starting Solr, the first log line should be more informative, such as

{code}
Welcome to Apache Solr™ version 7.0.0, running in standalone mode on port 8983 
from folder /Users/janhoy/git/lucene-solr/solr
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7826) Permission issues when creating cores with bin/solr

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511292#comment-15511292
 ] 

Jan Høydahl commented on SOLR-7826:
---

SOLR-9547

> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9547) Do not allow bin/solr start as root user (unless -force param specified)

[JIRA]

Jan Høydahl created SOLR-9547:
-

 Summary: Do not allow bin/solr start as root user (unless -force 
param specified)
 Key: SOLR-9547
 URL: https://issues.apache.org/jira/browse/SOLR-9547
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: scripts and tools
Reporter: Jan Høydahl


Spinoff from SOLR-7826

We should abort with a warning if user tries to start Solr as root, since this 
is simply not recommended and poses a security threat.
We should do the same for the "restart" option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-7457) Default doc values format should optimize for iterator access

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand reassigned LUCENE-7457:


Assignee: Adrien Grand

> Default doc values format should optimize for iterator access
> -
>
> Key: LUCENE-7457
> URL: https://issues.apache.org/jira/browse/LUCENE-7457
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Adrien Grand
>Priority: Blocker
> Fix For: master (7.0)
>
>
> In LUCENE-7407 we switched doc values consumption from random access API to 
> an iterator API, but nothing was done there to improve the codec.  We should 
> do that here.
> At a bare minimum we should fix the existing very-sparse case to be a true 
> iterator, and not wrapped with the silly legacy wrappers.
> I think we should also increase the threshold (currently 1%?) when we switch 
> from dense to sparse encoding.  This should fix LUCENE-7253, making merging 
> of sparse doc values efficient ("pay for what you use").
> I'm sure there are many other things to explore to let codecs "take 
> advantage" of the fact that they no longer need to offer random access to doc 
> values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7826) Permission issues when creating cores with bin/solr

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511249#comment-15511249
 ] 

Shawn Heisey commented on SOLR-7826:


bq. The question is if we should discourage and warn about starting solr as 
root as well, since this is not recommended?

I would say yes to this.  I'd even go a little bit further.  The script should 
refuse to run (without the force option) any command that creates or modifies 
filesystem data -- but only if Solr also writes to the same filesystem 
location.  This would mean that creating collections in cloud mode and options 
like upconfig and downconfig would be perfectly acceptable to run as root.

> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+136) - Build # 17869 - Still unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17869/
Java: 32bit/jdk-9-ea+136 -client -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testUpdateStream

Error Message:
Error from server at http://127.0.0.1:33671/solr: collection already exists: 
destinationCollection

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:33671/solr: collection already exists: 
destinationCollection
at 
__randomizedtesting.SeedInfo.seed([C38F5FC45BF89972:BC89BBE52F06422]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:608)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:415)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:367)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1280)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testUpdateStream(StreamExpressionTest.java:2700)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:535)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Assigned] (SOLR-6677) reduce logging during Solr startup

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-6677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl reassigned SOLR-6677:
-

Assignee: Jan Høydahl  (was: Noble Paul)

> reduce logging during Solr startup
> --
>
> Key: SOLR-6677
> URL: https://issues.apache.org/jira/browse/SOLR-6677
> Project: Solr
>  Issue Type: Bug
>Reporter: Noble Paul
>Assignee: Jan Høydahl
> Attachments: SOLR-6677.patch
>
>
> most of what is printed is neither helpful nor useful. It's just noise



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9450) Link to online Javadocs instead of distributing with binary download

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-9450:
--
Attachment: SOLR-9450.patch

Ok, attaching patch with separate {{solr.javadoc.url}} property.

When generating index.html in {{online-link.xsl}} we check for the text 
"SNAPSHOT" in {{$version}} and if not present we generate a link to whatever 
URL is in the property.

Was this how you imagined it, Uwe?

> Link to online Javadocs instead of distributing with binary download
> 
>
> Key: SOLR-9450
> URL: https://issues.apache.org/jira/browse/SOLR-9450
> Project: Solr
>  Issue Type: Sub-task
>  Components: Build
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9450.patch, SOLR-9450.patch, SOLR-9450.patch
>
>
> Spinoff from SOLR-6806. This sub task will replace the contents of {{docs}} 
> in the binary download with a link to the online JavaDocs. The build should 
> make sure to generate a link to the correct version. I believe this is the 
> correct tamplate: http://lucene.apache.org/solr/6_2_0/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511196#comment-15511196
 ] 

Pushkar Raste edited comment on SOLR-9546 at 9/21/16 9:16 PM:
--

Got you.
I will fix the  {{Long getLong(String param, Long def)}} method only. It is not 
as bad as initially thought.

I don't even think that method is needed. Calling {{Long getLong(String 
param)}} would do the same thing, won't it?


was (Author: praste):
Got you.
I will fix the  {{Long getLong(String param, Long def)}} method only. It is not 
as bad as initially thought

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511196#comment-15511196
 ] 

Pushkar Raste commented on SOLR-9546:
-

Got you.
I will fix the  {{Long getLong(String param, Long def)}} method only. It is not 
as bad as initially thought

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511162#comment-15511162
 ] 

Yonik Seeley commented on SOLR-9546:


bq. That was just one example. Check getBool()

I was responding to "unnecessary boxing".  For getBool for example, has both a 
boxed and primitive version:
The boxed version is so you can tell if a value was actually present, and the 
primitive version can be used if you provide the primitive default.
{code}
  /** Returns the Boolean value of the param, or null if not set */
  public Boolean getBool(String param) {
String val = get(param);
return val==null ? null : StrUtils.parseBool(val);
  }

  /** Returns the boolean value of the param, or def if not set */
  public boolean getBool(String param, boolean def) {
String val = get(param);
return val==null ? def : StrUtils.parseBool(val);
  }
{code}

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8395) query-time join (with scoring) for single value numeric fields

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated SOLR-8395:
---
Attachment: SOLR-8395.patch

Back to work. Meanwhile LUCENE-7418 nuked legacy numerics from {{JoinUtil}}. 
This patch pulls it back into {{ScoreJoinQParserPlugin}}, however it requires 
to expose some join internals on public. 
I understand that having points in Solr is better, but is there anything 
preventing from forgiving  such approach? 

> query-time join (with scoring) for single value numeric fields
> --
>
> Key: SOLR-8395
> URL: https://issues.apache.org/jira/browse/SOLR-8395
> Project: Solr
>  Issue Type: Improvement
>  Components: search
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
>Priority: Minor
>  Labels: easytest, features, newbie, starter
> Fix For: 5.5
>
> Attachments: SOLR-8395.patch, SOLR-8395.patch, SOLR-8395.patch, 
> SOLR-8395.patch, SOLR-8395.patch
>
>
> since LUCENE-5868 we have an opportunity to improve SOLR-6234 to make it join 
> int and long fields. I suppose it's worth to add "simple" test in Solr 
> NoScore suite. 
> * Alongside with that we can set _multipleValues_ parameters giving 
> _fromField_ cardinality declared in schema;



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushkar Raste updated SOLR-9546:

Comment: was deleted

(was: That was just one example. Check {{getBool()}}, {{getFieldBool()}} 
methods those have the exact same problem, and there are many more.

I am not sure which way we should go (primitive vs Wrapped types) but I am 
inclined towards primitive types.)

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7452) improve exception message: child query must only match non-parent docs, but parent docID=180314...

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511152#comment-15511152
 ] 

Michael McCandless commented on LUCENE-7452:


+1, thanks [~mkhludnev]!

> improve exception message: child query must only match non-parent docs, but 
> parent docID=180314...
> --
>
> Key: LUCENE-7452
> URL: https://issues.apache.org/jira/browse/LUCENE-7452
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 6.2
>Reporter: Mikhail Khludnev
>Priority: Minor
> Attachments: LUCENE-7452.patch
>
>
> when parent filter intersects with child query the exception exposes internal 
> details: docnum and scorer class. I propose an exception message to suggest 
> to execute a query intersecting them both. There is an opinion to add this  
> suggestion in addition to existing details. 
> My main concern against is, when index is constantly updated even SOLR-9582 
> allows to search for docnum it would be like catching the wind, also think 
> about cloud case. But, user advised with executing query intersection can 
> catch problem documents even if they occurs sporadically.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: GNU licenses inside licenses/*

[Chris Hostetter]

yeah, that seems pretty definitive ... hack away.


: Date: Wed, 21 Sep 2016 22:50:15 +0200
: From: Dawid Weiss 
: Reply-To: dev@lucene.apache.org
: To: "dev@lucene.apache.org" 
: Subject: Re: GNU licenses inside licenses/*
: 
: Browsed the archives of legal-discuss knowing the problem must have
: surfaced before. Found a lot of legal talk and got a major headache
: from reading it, but the conclusion seems to be here on "resolved"
: legal ASF cases:
: 
: http://www.apache.org/legal/resolved.html#mutually-exclusive
: 
: For those impatient (sorry for caps, copy-paste):
: 
: HOW SHOULD WORKS FOR WHICH MULTIPLE MUTUALLY EXCLUSIVE LICENSES ARE
: AVAILABLE BE HANDLED?
: 
: When including that work's licensing, state which license is being
: used and include only the license that you have chosen. Prefer
: Category A to Category B to Category X. You don't need to modify the
: work itself if, for example, it mentions the various licensing options
: in the source headers.
: 
: 
: On Wed, Sep 21, 2016 at 8:52 PM, Chris Hostetter
:  wrote:
: >
: > : then, who looks at those files anyway... Let's leave it as is if Chris
: > : says there was a discussion about it in the past.
: >
: > I could be wrong ... it's not like i looked very hard in the archives to
: > try and find any such discussion ... It's possible my mind is just filling
: > in the gaps with my own opinions.
: >
: > The PMC and/or Folks who care about this sort of thing might want to bring
: > it up with legal-discuss and figure out what the "right" answer is.
: >
: >
: >
: > -Hoss
: > http://www.lucidworks.com/
: >
: > -
: > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
: > For additional commands, e-mail: dev-h...@lucene.apache.org
: >
: 
: -
: To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
: For additional commands, e-mail: dev-h...@lucene.apache.org
: 
: 

-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Paul Elschot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511145#comment-15511145
 ] 

Paul Elschot edited comment on LUCENE-7453 at 9/21/16 8:56 PM:
---

bq. But the seg examples you have still have docid, just with seg prepended. It 
still has the problem that it uses "id", when id means identifier,

This is meant as an identifier for a document within a segment; in a segment 
this identifier is permanent. There may be another identifier in a document 
field, but that is irrelevant here.

For compound readers there are multiple segments, and also in that case adding 
seg to the name is correct.



was (Author: paul.elsc...@xs4all.nl):
bq. But the seg examples you have still have docid, just with seg prepended. It 
still has the problem that it uses "id", when id means identifier,

This is meant as an identifier for a document within a segment; in a segment 
this identifier is permanent, and the only one.

For compound readers there are multiple segments, and also in that case adding 
seg to the name is correct.


> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Paul Elschot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511145#comment-15511145
 ] 

Paul Elschot commented on LUCENE-7453:
--

bq. But the seg examples you have still have docid, just with seg prepended. It 
still has the problem that it uses "id", when id means identifier,

This is meant as an identifier for a document within a segment; in a segment 
this identifier is permanent, and the only one.

For compound readers there are multiple segments, and also in that case adding 
seg to the name is correct.


> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511132#comment-15511132
 ] 

Pushkar Raste edited comment on SOLR-9546 at 9/21/16 8:49 PM:
--

That was just one example. Check {{getBool()}}, {{getFieldBool()}} methods 
those have the exact same problem, and there are many more.

I am not sure which way we should go (primitive vs Wrapped types) but I am 
inclined towards primitive types.


was (Author: praste):
That was just one example check {{getBool()}}, {{getFieldBool()}} methods those 
have the exact same problem, and there are many more.

I am not sure which way we should go (primitive vs Wrapped types) but I am 
inclined towards primitive types.

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: GNU licenses inside licenses/*

[Dawid Weiss]

Browsed the archives of legal-discuss knowing the problem must have
surfaced before. Found a lot of legal talk and got a major headache
from reading it, but the conclusion seems to be here on "resolved"
legal ASF cases:

http://www.apache.org/legal/resolved.html#mutually-exclusive

For those impatient (sorry for caps, copy-paste):

HOW SHOULD WORKS FOR WHICH MULTIPLE MUTUALLY EXCLUSIVE LICENSES ARE
AVAILABLE BE HANDLED?

When including that work's licensing, state which license is being
used and include only the license that you have chosen. Prefer
Category A to Category B to Category X. You don't need to modify the
work itself if, for example, it mentions the various licensing options
in the source headers.


On Wed, Sep 21, 2016 at 8:52 PM, Chris Hostetter
 wrote:
>
> : then, who looks at those files anyway... Let's leave it as is if Chris
> : says there was a discussion about it in the past.
>
> I could be wrong ... it's not like i looked very hard in the archives to
> try and find any such discussion ... It's possible my mind is just filling
> in the gaps with my own opinions.
>
> The PMC and/or Folks who care about this sort of thing might want to bring
> it up with legal-discuss and figure out what the "right" answer is.
>
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511132#comment-15511132
 ] 

Pushkar Raste commented on SOLR-9546:
-

That was just one example check {{getBool()}}, {{getFieldBool()}} methods those 
have the exact same problem, and there are many more.

I am not sure which way we should go (primitive vs Wrapped types) but I am 
inclined towards primitive types.

> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9527) Solr RESTORE api doesn't distribute the replicas uniformly

[Stephen Lewis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Lewis updated SOLR-9527:

Attachment: SOLR-9527.patch

This update removes some extra logging I'd meant to rip and passes through the 
createNodeSet param from the request to the function call.

> Solr RESTORE api doesn't distribute the replicas uniformly 
> ---
>
> Key: SOLR-9527
> URL: https://issues.apache.org/jira/browse/SOLR-9527
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.1
>Reporter: Hrishikesh Gadre
> Attachments: SOLR-9527.patch, SOLR-9527.patch, SOLR-9527.patch
>
>
> Please refer to this email thread for details,
> http://lucene.markmail.org/message/ycun4x5nx7lwj5sk?q=solr+list:org%2Eapache%2Elucene%2Esolr-user+order:date-backward=1



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7826) Permission issues when creating cores with bin/solr

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1553#comment-1553
 ] 

Jan Høydahl commented on SOLR-7826:
---

Thanks for the discussion and the contributions. Binoy/Shawn, I hope you don't 
feel bad that I did not use your approach/patch. I thought it was a bit 
overkill. Since the installer script by default creates a "solr" user for 
starting solr, it would be very uncommon for Solr to be run as root, and 
absolutely not something we would recommend. So now we always warn.

The question is if we should discourage and warn about *starting* solr as root 
as well, since this is not recommended?

> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-7826) Permission issues when creating cores with bin/solr

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-7826.
---
Resolution: Fixed

> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511103#comment-15511103
 ] 

Yonik Seeley commented on SOLR-9546:


Hmmm, that's weird... the original methods did use primitives.
Looking at the current class, there *is* still the original version:

{code}
  /** Returns the long value of the param, or def if not set */
  public long getLong(String param, long def) {
String val = get(param);
try {
  return val == null ? def : Long.parseLong(val);
} catch (Exception ex) {
  throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, 
ex.getMessage(), ex);
}
  }
{code}

But it looks like the version you reference was added more recently as part of 
SOLR-6485
Anyway, I searched for usages, and that version is only used once (as part of 
the issue above it looks like)


> There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class
> --
>
> Key: SOLR-9546
> URL: https://issues.apache.org/jira/browse/SOLR-9546
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Pushkar Raste
>Priority: Minor
>
> Here is an excerpt 
> {code}
>   public Long getLong(String param, Long def) {
> String val = get(param);
> try {
>   return val== null ? def : Long.parseLong(val);
> }
> catch( Exception ex ) {
>   throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
> ex.getMessage(), ex );
> }
>   }
> {code}
> {{Long.parseLong()}} returns a primitive type but since method expect to 
> return a {{Long}}, it needs to be wrapped. There are many more method like 
> that. We might be creating a lot of unnecessary objects here.
> I am not sure if JVM catches upto it and somehow optimizes it if these 
> methods are called enough times (or may be compiler does some modifications 
> at compile time)
> Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511085#comment-15511085
 ] 

Dawid Weiss commented on LUCENE-7453:
-

Hoss, consider the possibilities of using non-ascii unicode letters (valid Java 
identifiers). Perfectly fine to call it {{docλ}}. Or even better, just {{ℵ}}. 
{{ℵSet}}. {{ℵIterator}}. Think of it, wouldn't that be truly unique.

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7826) Permission issues when creating cores with bin/solr

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511075#comment-15511075
 ] 

ASF subversion and git services commented on SOLR-7826:
---

Commit 6b10283765cf015aad59f054919134662d060c3b in lucene-solr's branch 
refs/heads/branch_6x from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6b10283 ]

SOLR-7826: Refuse "bin/solr create" if run as root, unless -force is specified

(cherry picked from commit 7561461)


> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (32bit/jdk1.8.0_102) - Build # 1770 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1770/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 1) Thread[id=2730, 
name=TEST-CollectionsAPIDistributedZkTest.test-seed#[F2E44CC3B55EEF6B]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
2) Thread[id=2801, name=zkCallback-420-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)3) Thread[id=2729, 
name=TEST-CollectionsAPIDistributedZkTest.test-seed#[F2E44CC3B55EEF6B]-SendThread(127.0.0.1:45437),
 state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at 
java.lang.Thread.sleep(Native Method) at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
 at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
4) Thread[id=2731, name=zkCallback-420-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)5) Thread[id=2800, 
name=zkCallback-420-thread-2, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=2730, 
name=TEST-CollectionsAPIDistributedZkTest.test-seed#[F2E44CC3B55EEF6B]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
   2) Thread[id=2801, name=zkCallback-420-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at 

[jira] [Comment Edited] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511065#comment-15511065
 ] 

Mikhail Khludnev edited comment on LUCENE-7453 at 9/21/16 8:24 PM:
---

Whatever you name it, let's have _dixi_ in code! Please!


was (Author: mkhludnev):
Whatever you call name, let's have _dixi_ in code! Please!

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511065#comment-15511065
 ] 

Mikhail Khludnev commented on LUCENE-7453:
--

Whatever you call name, let's have _dixi_ in code! Please!

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7826) Permission issues when creating cores with bin/solr

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511067#comment-15511067
 ] 

ASF subversion and git services commented on SOLR-7826:
---

Commit 7561461f738a447856bb93b0a847b0200fff4c9c in lucene-solr's branch 
refs/heads/master from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=7561461 ]

SOLR-7826: Refuse "bin/solr create" if run as root, unless -force is specified


> Permission issues when creating cores with bin/solr
> ---
>
> Key: SOLR-7826
> URL: https://issues.apache.org/jira/browse/SOLR-7826
> Project: Solr
>  Issue Type: Improvement
>Reporter: Shawn Heisey
>Assignee: Jan Høydahl
>Priority: Minor
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-7826.patch, SOLR-7826.patch
>
>
> Ran into an interesting situation on IRC today.
> Solr has been installed as a service using the shell script 
> install_solr_service.sh ... so it is running as an unprivileged user.
> User is running "bin/solr create" as root.  This causes permission problems, 
> because the script creates the core's instanceDir with root ownership, then 
> when Solr is instructed to actually create the core, it cannot create the 
> dataDir.
> Enhancement idea:  When the install script is used, leave breadcrumbs 
> somewhere so that the "create core" section of the main script can find it 
> and su to the user specified during install.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9546) There is a lot of unnecessary boxing/unboxing going on in {{SolrParams}} class

[Pushkar Raste (JIRA)]

Pushkar Raste created SOLR-9546:
---

 Summary: There is a lot of unnecessary boxing/unboxing going on in 
{{SolrParams}} class
 Key: SOLR-9546
 URL: https://issues.apache.org/jira/browse/SOLR-9546
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Pushkar Raste
Priority: Minor


Here is an excerpt 

{code}
  public Long getLong(String param, Long def) {
String val = get(param);
try {
  return val== null ? def : Long.parseLong(val);
}
catch( Exception ex ) {
  throw new SolrException( SolrException.ErrorCode.BAD_REQUEST, 
ex.getMessage(), ex );
}
  }
{code}

{{Long.parseLong()}} returns a primitive type but since method expect to return 
a {{Long}}, it needs to be wrapped. There are many more method like that. We 
might be creating a lot of unnecessary objects here.

I am not sure if JVM catches upto it and somehow optimizes it if these methods 
are called enough times (or may be compiler does some modifications at compile 
time)


Let me know if I am thinking of some premature optimization



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Ryan Ernst (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511049#comment-15511049
 ] 

Ryan Ernst commented on LUCENE-7453:


bq. this is probably a multicriteria optimisation problem with pareto-optimal 
set of solutions, rather than a single one...

That is why I'm ok with "docIndex" here, it just isn't my favorite. But it is 
still better than docid.

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511047#comment-15511047
 ] 

Hoss Man commented on LUCENE-7453:
--

We could just call it the {{huperDuperEphemeralPositionInIndex}}

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Ryan Ernst (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511042#comment-15511042
 ] 

Ryan Ernst commented on LUCENE-7453:


bq. I don't like overloading index for this, especially in the class names, so 
for now I'd prefer the segment variants in second column.

But the seg examples you have still have docid, just with seg prepended. It 
still has the problem that it uses "id", when id means identifier, which is 
usually an opaque string. {{docnum}} to me is still the best, the document 
number in the segment (where "in the segment is implied", although if we want 
SegDocNum, I guess it'd be ok, just more wordy).

bq. Anyway, we could use the opportunity to shorten some of the longer names.

+1

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15511028#comment-15511028
 ] 

Dawid Weiss commented on LUCENE-7453:
-

bq. [...] i meant the overloading of "index", because it is used in so many 
ways (inverted index and array index, which are obviously similar, but very 
different things, one is a collection of data structures in files, and the 
other is a number).

That the term is overloaded with meanings doesn't mean this particular use case 
isn't appropriate. 

Looking at Paul's table the {{index}} column still looks semantically most 
clear to my eyes. What I like in particular is that there is no notion of the 
"index" belonging to a segment (as {{SegDocIdSet}} would imply). Rather, like I 
said before, the doc index is a logical index of a document within an index 
reader (whether it's a leaf reader or a composite doesn't matter). It nicely 
fits in loops created for {{reader.maxDoc}}, for example.

This said, everyone will have their favorites; this is probably a multicriteria 
optimisation problem with pareto-optimal set of solutions, rather than a single 
one...


> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9508) Install script should check existence of tools, and add option to NOT start service

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-9508.
---
Resolution: Fixed

> Install script should check existence of tools, and add option to NOT start 
> service
> ---
>
> Key: SOLR-9508
> URL: https://issues.apache.org/jira/browse/SOLR-9508
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: scripts and tools
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9508.patch
>
>
> The {{install_solr_service.sh}} script should exit cleanly if tools like 
> {{tar}}, {{unzip}}, {{service}} or {{java}} are not available.
> Also, add a new switch {{-n}} to skip starting the service after 
> installation, which will make it easier to script installations which will 
> want to modify {{/etc/default/solr.in.sh}} before starting the service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9508) Install script should check existence of tools, and add option to NOT start service

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510979#comment-15510979
 ] 

ASF subversion and git services commented on SOLR-9508:
---

Commit 33874f9ece1bafc2952f546d22c43998cbd43806 in lucene-solr's branch 
refs/heads/branch_6x from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=33874f9 ]

SOLR-9508: Install script should check existence of tools, and add option to 
NOT start service

(cherry picked from commit b894ab2)


> Install script should check existence of tools, and add option to NOT start 
> service
> ---
>
> Key: SOLR-9508
> URL: https://issues.apache.org/jira/browse/SOLR-9508
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: scripts and tools
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9508.patch
>
>
> The {{install_solr_service.sh}} script should exit cleanly if tools like 
> {{tar}}, {{unzip}}, {{service}} or {{java}} are not available.
> Also, add a new switch {{-n}} to skip starting the service after 
> installation, which will make it easier to script installations which will 
> want to modify {{/etc/default/solr.in.sh}} before starting the service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9508) Install script should check existence of tools, and add option to NOT start service

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510971#comment-15510971
 ] 

ASF subversion and git services commented on SOLR-9508:
---

Commit b894ab292d3f319d03f375fc04376fbee6760b3f in lucene-solr's branch 
refs/heads/master from [~janhoy]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b894ab2 ]

SOLR-9508: Install script should check existence of tools, and add option to 
NOT start service


> Install script should check existence of tools, and add option to NOT start 
> service
> ---
>
> Key: SOLR-9508
> URL: https://issues.apache.org/jira/browse/SOLR-9508
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: scripts and tools
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9508.patch
>
>
> The {{install_solr_service.sh}} script should exit cleanly if tools like 
> {{tar}}, {{unzip}}, {{service}} or {{java}} are not available.
> Also, add a new switch {{-n}} to skip starting the service after 
> installation, which will make it easier to script installations which will 
> want to modify {{/etc/default/solr.in.sh}} before starting the service.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Paul Elschot (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510965#comment-15510965
 ] 

Paul Elschot commented on LUCENE-7453:
--

I tried an alternative that adds an variation of segment wherever docID is used 
in some form.

Here is an overview of renaming possibilities for core/src/java, in three 
column python strings.

The first column contains the current name, the second column a segment 
variant, the third column an index variant.
Please assume an appropriate amount of question marks (??) in the second and 
third columns.

{code}
classFileRenames = """

DocIdSet SegDocIdSet  DocIndexSet
DocIdSetIterator SegDocIdSetIterator  
DocIndexIterator
ConjunctionDISI  ConjunctionSegDisi   
ConjunctionDixi
DisjunctionDISIApproximation DisjunctionSegDisiApproximation  
DisjunctionDixiApproximation
DisiPriorityQueueSegDisiPriorityQueue 
DixiPriorityQueue
DisiWrapper  SegDisiWrapper   DixiWrapper
FilteredDocIdSetIterator FilteredSegDisi  FilteredDixi
DocIdSetBuilder  SegDocIdSetBuilder   
DocIndexSetBuilder
RoaringDocIdSet  RoaringSegDocIdSet   
RoaringDocIndexSet
IntArrayDocIdSet IntArraySegDocIdSet  
IntArrayDocIndexSet
NotDocIdSet  NotSegDocIdSet   NotDocIndexSet
BitDocIdSet  BitSegDocIdSet   BitDocIndexSet
DocIdsWriter SegDocIdsWriter  
DocIndexesWriter
DocIdMerger  SegDocIdMerger   DocIndexMerger
"""

identifierRenames = classFileRenames + """

TwoPhaseIteratorAsDocIdSetIterator TwoPhaseIteratorAsSegDocIdSetIterator 
TwoPhaseIteratorAsDocIndexIterator
BitSetConjunctionDISI  BitSetConjunctionDisi 
BitSetConjunctionDisi
IntArrayDocIdSetIterator   IntArraySegDocIdSetIterator   
IntArrayDocIndexIterator

asDocIdSetIterator asSegDocIdSetIterator 
asDocIndexIterator
getDocId   getSegDocId   
getDocIndex
docID  sdocID
docIndex

docID  sdocIDdocIdx
docId  sdocIddocIdx
docIDs sdocIDs   docIdxs
docIds sdocIds   docIdxs
disi   sdisi dixi
docIdSet   sDocIdSet 
docIndexSet

"""
{code}

(The identifiers here are for local classes, methods and variables.)

I don't like overloading index for this, especially in the class names, so for 
now I'd prefer the segment variants in second column.

Anyway, we could use the opportunity to shorten some of the longer names.


> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (32bit/jdk1.8.0_102) - Build # 17868 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17868/
Java: 32bit/jdk1.8.0_102 -server -XX:+UseParallelGC

4 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamExpressionTest

Error Message:
11 threads leaked from SUITE scope at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest: 1) 
Thread[id=1637, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at java.lang.Thread.sleep(Native 
Method) at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.lang.Thread.run(Thread.java:745)2) Thread[id=1652, 
name=Connection evictor, state=TIMED_WAITING, group=TGRP-StreamExpressionTest]  
   at java.lang.Thread.sleep(Native Method) at 
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.lang.Thread.run(Thread.java:745)3) Thread[id=2130, 
name=zkCallback-517-thread-2, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at sun.misc.Unsafe.park(Native Method) 
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)4) Thread[id=1650, 
name=TextLogitSolrStream-518-thread-2, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at sun.misc.Unsafe.park(Native Method) 
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)5) Thread[id=1639, 
name=TEST-StreamExpressionTest.testBasicTextLogitStream-seed#[E3C4FB554FA3A976]-EventThread,
 state=WAITING, group=TGRP-StreamExpressionTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
6) Thread[id=1640, name=zkCallback-517-thread-1, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at sun.misc.Unsafe.park(Native Method) 
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)7) Thread[id=1638, 
name=TEST-StreamExpressionTest.testBasicTextLogitStream-seed#[E3C4FB554FA3A976]-SendThread(127.0.0.1:43396),
 state=TIMED_WAITING, group=TGRP-StreamExpressionTest] at 
java.lang.Thread.sleep(Native Method) at 
org.apache.zookeeper.client.StaticHostProvider.next(StaticHostProvider.java:101)
 at 
org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:940)
 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1003)
8) Thread[id=1649, name=TextLogitSolrStream-518-thread-1, state=TIMED_WAITING, 
group=TGRP-StreamExpressionTest] at sun.misc.Unsafe.park(Native Method) 
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 

[jira] [Updated] (SOLR-9258) Optimizing, storing and deploying AI models with Streaming Expressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9258:
-
Attachment: SOLR-9258.patch

New patch which adds the cacheMillis param to the model() expression

> Optimizing, storing and deploying AI models with Streaming Expressions
> --
>
> Key: SOLR-9258
> URL: https://issues.apache.org/jira/browse/SOLR-9258
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: 6.2
>
> Attachments: ModelCache.java, ModelCache.java, SOLR-9258.patch, 
> SOLR-9258.patch, SOLR-9258.patch, SOLR-9258.patch
>
>
> This ticket describes a framework for *optimizing*, *storing* and *deploying* 
> AI models within the Streaming Expression framework.
> *Optimizing*
> [~caomanhdat], has contributed SOLR-9252 which provides *Streaming 
> Expressions* for both feature selection and optimization of a logistic 
> regression text classifier. SOLR-9252 also provides a great working example 
> of *optimization* of a machine learning model using an in-place parallel 
> iterative algorithm.
> *Storing*
> Both features and optimized models can be stored in SolrCloud collections 
> using the update expression. Using [~caomanhdat]'s example in SOLR-9252, the 
> pseudo code for storing features would be:
> {code}
> update(featuresCollection, 
>featuresSelection(collection1, 
> id="myFeatures", 
> q="*:*",  
> field="tv_text", 
> outcome="out_i", 
> positiveLabel=1, 
> numTerms=100))
> {code}  
> The id field can be added to the featureSelection expression so that features 
> can be later retrieved from the collection it's stored in.
> *Deploying*
> With the introduction of the topic() expression, SolrCloud can be treated as 
> a distributed message queue. This messaging capability can  be used to deploy 
> models and process data through the models.
> To implement this approach a classify() function can be created that uses a 
> topic() function to return both the model and the data to be classified:
> The pseudo code looks like this:
> {code}
> classify(topic(models, q="modelID", fl="features, weights"),
>  topic(emails, q="*:*", fl="id, body", rows="500", version="3232323"))
> {code}
> In the example above the classify() function uses the topic() function to 
> retrieve the model. Each time there is an update to the model in the index, 
> the topic() expression will automatically read the new model.
> The topic function() is also used to pull in the data set that is being 
> classified. Notice the *version* parameter. This will be added to the topic 
> function to support pulling results from a specific version number (jira 
> ticket to follow).
> With this approach both the model and the data to process through the model 
> are treated as messages in a message queue.
> The daemon function can be used to send the classify function to Solr where 
> it will be run in the background. The pseudo code looks like this:
> {code}
> daemon(...,
>  update(classifiedEmails, 
>  classify(topic(models, q="modelID", fl="features, weights"),
>   topic(emails, q="*:*", fl="id, fl, body", 
> rows="500", version="3232323"
> {code}
> In this scenario the daemon will run the classify function repeatedly in the 
> background. With each run the topic() functions will re-pull the model if the 
> model has been updated. It will also pull a new set of emails to be 
> classified. The classified emails can be stored in another SolrCloud 
> collection using the update() function.
> Using this approach emails can be classified in batches. The daemon can 
> continue to run even after all all the emails have been classified. New 
> emails added to the emails collections will then be automatically classified 
> when they enter the index.
> Classification can be done in parallel once SOLR-9240 is completed. This will 
> allow topic() results to be partitioned across worker nodes so they can be 
> processed in parallel. The pseudo code for this is:
> {code}
> parallel(workerCollection, worker="20", ...,
>  daemon(...,
>update(classifiedEmails, 
>classify(topic(models, q="modelID", fl="features, 
> weights", partitionKeys="none"),
> topic(emails, q="*:*", fl="id, fl, body", 
> rows="500", version="3232323", partitionKeys="id")
> {code}
> The code above sends a daemon to 20 workers, which will each classify a 
> partition of records 

[jira] [Updated] (LUCENE-7452) improve exception message: child query must only match non-parent docs, but parent docID=180314...

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated LUCENE-7452:
-
Attachment: LUCENE-7452.patch

What about this? 
{panel}
java.lang.IllegalStateException: Child query must not match same docs with 
parent filter. Combine them as must clauses (+) to find a problem doc. 
docId=23, class org.apache.lucene.search.DisjunctionSumScorer
{panel}

{panel}
java.lang.IllegalStateException: Parent query must not match any docs beside 
parent filter. Combine them as must (+) and must-not (-) clauses to find a 
problem doc. docID=12
{panel}


> improve exception message: child query must only match non-parent docs, but 
> parent docID=180314...
> --
>
> Key: LUCENE-7452
> URL: https://issues.apache.org/jira/browse/LUCENE-7452
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 6.2
>Reporter: Mikhail Khludnev
>Priority: Minor
> Attachments: LUCENE-7452.patch
>
>
> when parent filter intersects with child query the exception exposes internal 
> details: docnum and scorer class. I propose an exception message to suggest 
> to execute a query intersecting them both. There is an opinion to add this  
> suggestion in addition to existing details. 
> My main concern against is, when index is constantly updated even SOLR-9582 
> allows to search for docnum it would be like catching the wind, also think 
> about cloud case. But, user advised with executing query intersection can 
> catch problem documents even if they occurs sporadically.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8487) Add CommitStream to Streaming API and Streaming Expressions

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510877#comment-15510877
 ] 

Joel Bernstein commented on SOLR-8487:
--

+1 looks good

> Add CommitStream to Streaming API and Streaming Expressions
> ---
>
> Key: SOLR-8487
> URL: https://issues.apache.org/jira/browse/SOLR-8487
> Project: Solr
>  Issue Type: New Feature
>Affects Versions: 6.0
>Reporter: Jason Gerlowski
>Priority: Minor
> Fix For: 6.0
>
> Attachments: SOLR-8487.patch, SOLR-8487.patch
>
>
> (Paraphrased from Joel's idea/suggestions in the comments of SOLR-7535).
> With SOLR-7535, users can now index documents/tuples using an UpdateStream.  
> However, there's no way currently using the Streaming API to force a commit 
> on the collection that received these updates.
> The purpose of this ticket is to add a CommitStream, which can be used to 
> trigger commit(s) on a given collection.
> The proposed usage/behavior would look a little bit like:
> {{commit(collection, parallel(update(search()))}}
> Note that...
> 1.) CommitStream has a positional collection parameter, to indicate which 
> collection to commit on. (Alternatively, it could recurse through 
> {{children()}} nodes until it finds the UpdateStream, and then retrieve the 
> collection from the UpdateStream).
> 2.) CommitStream forwards all tuples received by an underlying, wrapped 
> stream.
> 3.) CommitStream commits when the underlying stream emits its EOF tuple. 
> (Alternatively, it could commit every X tuples, based on a parameter).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-7452) improve exception message: child query must only match non-parent docs, but parent docID=180314...

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510875#comment-15510875
 ] 

Mikhail Khludnev edited comment on LUCENE-7452 at 9/21/16 7:12 PM:
---

What about this? 
{panel}
java.lang.IllegalStateException: Child query must not match same docs with 
parent filter. Combine them as must clauses \(+) to find a problem doc. 
docId=23, class org.apache.lucene.search.DisjunctionSumScorer
{panel}

{panel}
java.lang.IllegalStateException: Parent query must not match any docs beside 
parent filter. Combine them as must \(+) and must-not \(-) clauses to find a 
problem doc. docID=12
{panel}



was (Author: mkhludnev):
What about this? 
{panel}
java.lang.IllegalStateException: Child query must not match same docs with 
parent filter. Combine them as must clauses (+) to find a problem doc. 
docId=23, class org.apache.lucene.search.DisjunctionSumScorer
{panel}

{panel}
java.lang.IllegalStateException: Parent query must not match any docs beside 
parent filter. Combine them as must (+) and must-not (-) clauses to find a 
problem doc. docID=12
{panel}


> improve exception message: child query must only match non-parent docs, but 
> parent docID=180314...
> --
>
> Key: LUCENE-7452
> URL: https://issues.apache.org/jira/browse/LUCENE-7452
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/search
>Affects Versions: 6.2
>Reporter: Mikhail Khludnev
>Priority: Minor
> Attachments: LUCENE-7452.patch
>
>
> when parent filter intersects with child query the exception exposes internal 
> details: docnum and scorer class. I propose an exception message to suggest 
> to execute a query intersecting them both. There is an opinion to add this  
> suggestion in addition to existing details. 
> My main concern against is, when index is constantly updated even SOLR-9582 
> allows to search for docnum it would be like catching the wind, also think 
> about cloud case. But, user advised with executing query intersection can 
> catch problem documents even if they occurs sporadically.  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8975) SolrClient setters should be deprecated in favor of SolrClientBuilder methods

[Jason Gerlowski (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510829#comment-15510829
 ] 

Jason Gerlowski commented on SOLR-8975:
---

I've started work on this in earnest, after an embarassingly long delay.  As a 
change, it's reminiscent of SOLR-8097.  The change is simple conceptually, but 
involves many smaller changes to the tests, due to their use of SolrClients.

For instance, many tests choose a ResponseParser randomly, often changing the 
type frequently (each iteration through a for-loop) for added randomness.  In a 
"immutable-SolrClient-world" this approach isn't possible anymore, are at least 
it would require new clients to be created often.  This is relatively cheap for 
{{HttpSolrClient}}, but could prove to be prohibitive for {{CloudSolrClient}} 
usage.

I plan on pushing up a sample patch once I work through the work related to 
{{HttpSolrClient}}, so changes/suggestions can be discussed before I slog 
through the rest of the SolrClients.

> SolrClient setters should be deprecated in favor of SolrClientBuilder methods
> -
>
> Key: SOLR-8975
> URL: https://issues.apache.org/jira/browse/SOLR-8975
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Reporter: Jason Gerlowski
>Priority: Minor
>
> SOLR-8097 added a builder layer on top of each {{SolrClient}} implementation.
> Now that builders are in place for SolrClients, the setters used in each 
> SolrClient can be deprecated, and their functionality moved over to the 
> Builders.  This change brings a few benefits:
> - unifies SolrClient configuration under the new Builders.  It'll be nice to 
> have all the knobs, and levers used to tweak SolrClients available in a 
> single place (the Builders).
> - reduces SolrClient thread-safety concerns.  Currently, clients are mutable. 
>  Using some SolrClient setters can result in erratic and "trappy" behavior 
> when the clients are used across multiple threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: GNU licenses inside licenses/*

[Chris Hostetter]

: then, who looks at those files anyway... Let's leave it as is if Chris
: says there was a discussion about it in the past.

I could be wrong ... it's not like i looked very hard in the archives to 
try and find any such discussion ... It's possible my mind is just filling 
in the gaps with my own opinions.

The PMC and/or Folks who care about this sort of thing might want to bring 
it up with legal-discuss and figure out what the "right" answer is.



-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510820#comment-15510820
 ] 

Yonik Seeley commented on LUCENE-7453:
--


Indeed, "docIndex" can also be read as "an index of documents" (just like term 
index is an index of terms).
docOrd is another option, but I don't like it.

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Ryan Ernst (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510784#comment-15510784
 ] 

Ryan Ernst commented on LUCENE-7453:


bq. although I do dislike using such an overloaded term

Just to be perfectly clear about this statement, i meant the overloading of 
"index", because it is used in so many ways (inverted index and array index, 
which are obviously similar, but very different things, one is a collection of 
data structures in files, and the other is a number).

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9446) Leader failure after creating a freshly replicated index can send nodes into recovery even if index was not changed

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-9446.
--
   Resolution: Fixed
Fix Version/s: master (7.0)
   6.3

> Leader failure after creating a freshly replicated index can send nodes into 
> recovery even if index was not changed
> ---
>
> Key: SOLR-9446
> URL: https://issues.apache.org/jira/browse/SOLR-9446
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: Pushkar Raste
>Assignee: Noble Paul
>Priority: Minor
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9446.patch
>
>
>  We noticed this issue while migrating solr index from machines {{A1, A2 and 
> A3}} to {{B1, B2, B3}}. We followed following steps (and there were no 
> updates during the migration process).
> * Index had replicas on machines {{A1, A2, A3}}. Let's say {{A1}} was the 
> leader at the time
> * We added 3 more replicas {{B1, B2 and B3}}. These nodes synced with the by 
> replication. These fresh nodes do not have tlogs.
> * We shut down one of the old nodes ({{A3}}). 
> * We then shut down the leader ({{A1}})
> * New leader got elected (let's say {{A2}}) became the new leader
> * Leader asked all the replicas to sync with it
> * Fresh nodes (ones without tlogs), first tried PeerSync but since there was 
> no frame of reference, PeerSync failed and fresh nodes fail back on to try 
> replication 
> Although replication would not copy all the segments again, it seems like we 
> can short circuit sync to put nodes back in active state as soon as possible. 
> If in case freshly replicated index becomes leader for some reason, it can 
> still send nodes (both other freshly replicated indexes and old replicas) 
> into recovery. Here is the scenario
> * Freshly replicated becomes the leader.
> * New leader however asks all the replicas to sync with it.
> * Replicas (including old one) ask for versions from the leader, but the 
> leader has no update logs, hence replicas can not compute missing versions 
> and falls back to replication



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9446) Leader failure after creating a freshly replicated index can send nodes into recovery even if index was not changed

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510768#comment-15510768
 ] 

ASF subversion and git services commented on SOLR-9446:
---

Commit 8502995e3b1ce66db49be26b23a3fa3c169345a8 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8502995 ]

SOLR-9446: Leader failure after creating a freshly replicated index can send 
nodes into recovery even if index was not changed


> Leader failure after creating a freshly replicated index can send nodes into 
> recovery even if index was not changed
> ---
>
> Key: SOLR-9446
> URL: https://issues.apache.org/jira/browse/SOLR-9446
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: Pushkar Raste
>Assignee: Noble Paul
>Priority: Minor
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-9446.patch
>
>
>  We noticed this issue while migrating solr index from machines {{A1, A2 and 
> A3}} to {{B1, B2, B3}}. We followed following steps (and there were no 
> updates during the migration process).
> * Index had replicas on machines {{A1, A2, A3}}. Let's say {{A1}} was the 
> leader at the time
> * We added 3 more replicas {{B1, B2 and B3}}. These nodes synced with the by 
> replication. These fresh nodes do not have tlogs.
> * We shut down one of the old nodes ({{A3}}). 
> * We then shut down the leader ({{A1}})
> * New leader got elected (let's say {{A2}}) became the new leader
> * Leader asked all the replicas to sync with it
> * Fresh nodes (ones without tlogs), first tried PeerSync but since there was 
> no frame of reference, PeerSync failed and fresh nodes fail back on to try 
> replication 
> Although replication would not copy all the segments again, it seems like we 
> can short circuit sync to put nodes back in active state as soon as possible. 
> If in case freshly replicated index becomes leader for some reason, it can 
> still send nodes (both other freshly replicated indexes and old replicas) 
> into recovery. Here is the scenario
> * Freshly replicated becomes the leader.
> * New leader however asks all the replicas to sync with it.
> * Replicas (including old one) ask for versions from the leader, but the 
> leader has no update logs, hence replicas can not compute missing versions 
> and falls back to replication



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9446) Leader failure after creating a freshly replicated index can send nodes into recovery even if index was not changed

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510764#comment-15510764
 ] 

ASF subversion and git services commented on SOLR-9446:
---

Commit 15cee3141c160c38756ceed73bd1cd88002c01c9 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=15cee31 ]

SOLR-9446: Leader failure after creating a freshly replicated index can send 
nodes into recovery even if index was not changed


> Leader failure after creating a freshly replicated index can send nodes into 
> recovery even if index was not changed
> ---
>
> Key: SOLR-9446
> URL: https://issues.apache.org/jira/browse/SOLR-9446
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: Pushkar Raste
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-9446.patch
>
>
>  We noticed this issue while migrating solr index from machines {{A1, A2 and 
> A3}} to {{B1, B2, B3}}. We followed following steps (and there were no 
> updates during the migration process).
> * Index had replicas on machines {{A1, A2, A3}}. Let's say {{A1}} was the 
> leader at the time
> * We added 3 more replicas {{B1, B2 and B3}}. These nodes synced with the by 
> replication. These fresh nodes do not have tlogs.
> * We shut down one of the old nodes ({{A3}}). 
> * We then shut down the leader ({{A1}})
> * New leader got elected (let's say {{A2}}) became the new leader
> * Leader asked all the replicas to sync with it
> * Fresh nodes (ones without tlogs), first tried PeerSync but since there was 
> no frame of reference, PeerSync failed and fresh nodes fail back on to try 
> replication 
> Although replication would not copy all the segments again, it seems like we 
> can short circuit sync to put nodes back in active state as soon as possible. 
> If in case freshly replicated index becomes leader for some reason, it can 
> still send nodes (both other freshly replicated indexes and old replicas) 
> into recovery. Here is the scenario
> * Freshly replicated becomes the leader.
> * New leader however asks all the replicas to sync with it.
> * Replicas (including old one) ask for versions from the leader, but the 
> leader has no update logs, hence replicas can not compute missing versions 
> and falls back to replication



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9258) Optimizing, storing and deploying AI models with Streaming Expressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9258:
-
Attachment: SOLR-9258.patch

New patch with passing StreamExpressionTest for the Classify function. Also 
exercises the model function,

I'll add some code to test the model caching behavior shortly.

> Optimizing, storing and deploying AI models with Streaming Expressions
> --
>
> Key: SOLR-9258
> URL: https://issues.apache.org/jira/browse/SOLR-9258
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: 6.2
>
> Attachments: ModelCache.java, ModelCache.java, SOLR-9258.patch, 
> SOLR-9258.patch, SOLR-9258.patch
>
>
> This ticket describes a framework for *optimizing*, *storing* and *deploying* 
> AI models within the Streaming Expression framework.
> *Optimizing*
> [~caomanhdat], has contributed SOLR-9252 which provides *Streaming 
> Expressions* for both feature selection and optimization of a logistic 
> regression text classifier. SOLR-9252 also provides a great working example 
> of *optimization* of a machine learning model using an in-place parallel 
> iterative algorithm.
> *Storing*
> Both features and optimized models can be stored in SolrCloud collections 
> using the update expression. Using [~caomanhdat]'s example in SOLR-9252, the 
> pseudo code for storing features would be:
> {code}
> update(featuresCollection, 
>featuresSelection(collection1, 
> id="myFeatures", 
> q="*:*",  
> field="tv_text", 
> outcome="out_i", 
> positiveLabel=1, 
> numTerms=100))
> {code}  
> The id field can be added to the featureSelection expression so that features 
> can be later retrieved from the collection it's stored in.
> *Deploying*
> With the introduction of the topic() expression, SolrCloud can be treated as 
> a distributed message queue. This messaging capability can  be used to deploy 
> models and process data through the models.
> To implement this approach a classify() function can be created that uses a 
> topic() function to return both the model and the data to be classified:
> The pseudo code looks like this:
> {code}
> classify(topic(models, q="modelID", fl="features, weights"),
>  topic(emails, q="*:*", fl="id, body", rows="500", version="3232323"))
> {code}
> In the example above the classify() function uses the topic() function to 
> retrieve the model. Each time there is an update to the model in the index, 
> the topic() expression will automatically read the new model.
> The topic function() is also used to pull in the data set that is being 
> classified. Notice the *version* parameter. This will be added to the topic 
> function to support pulling results from a specific version number (jira 
> ticket to follow).
> With this approach both the model and the data to process through the model 
> are treated as messages in a message queue.
> The daemon function can be used to send the classify function to Solr where 
> it will be run in the background. The pseudo code looks like this:
> {code}
> daemon(...,
>  update(classifiedEmails, 
>  classify(topic(models, q="modelID", fl="features, weights"),
>   topic(emails, q="*:*", fl="id, fl, body", 
> rows="500", version="3232323"
> {code}
> In this scenario the daemon will run the classify function repeatedly in the 
> background. With each run the topic() functions will re-pull the model if the 
> model has been updated. It will also pull a new set of emails to be 
> classified. The classified emails can be stored in another SolrCloud 
> collection using the update() function.
> Using this approach emails can be classified in batches. The daemon can 
> continue to run even after all all the emails have been classified. New 
> emails added to the emails collections will then be automatically classified 
> when they enter the index.
> Classification can be done in parallel once SOLR-9240 is completed. This will 
> allow topic() results to be partitioned across worker nodes so they can be 
> processed in parallel. The pseudo code for this is:
> {code}
> parallel(workerCollection, worker="20", ...,
>  daemon(...,
>update(classifiedEmails, 
>classify(topic(models, q="modelID", fl="features, 
> weights", partitionKeys="none"),
> topic(emails, q="*:*", fl="id, fl, body", 
> rows="500", version="3232323", partitionKeys="id")
> {code}
> The code 

[jira] [Commented] (SOLR-9545) DataImportHandler throws NPE to logs when pk attribute is not present when delta query is used

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-9545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510753#comment-15510753
 ] 

Rafał Kuć commented on SOLR-9545:
-

Sure, will modify the code and provide a proper patch :) 

> DataImportHandler throws NPE to logs when pk attribute is not present when 
> delta query is used
> --
>
> Key: SOLR-9545
> URL: https://issues.apache.org/jira/browse/SOLR-9545
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 6.2.1
>Reporter: Rafał Kuć
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-9545.patch
>
>
> Hi, 
> Currently, when running a delta query from the Data Import Handler and pk 
> parameter is not specified Solr just logs NullPointerExeception, not 
> providing any information on what was expected. 
> Patch coming soon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9545) DataImportHandler throws NPE to logs when pk attribute is not present when delta query is used

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-9545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510749#comment-15510749
 ] 

Tomás Fernández Löbbe commented on SOLR-9545:
-

Thanks for the patch [~gro], If I understand correctly, this code is executed 
when requesting a delta update but a "deltaImportQuery" was not configured, 
correct? In that case, the exception message is not correct. Also, could you 
make the exception a SolrException instead of a plain RuntimeException? and 
ideally add a unit test.

> DataImportHandler throws NPE to logs when pk attribute is not present when 
> delta query is used
> --
>
> Key: SOLR-9545
> URL: https://issues.apache.org/jira/browse/SOLR-9545
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - DataImportHandler
>Affects Versions: 6.2.1
>Reporter: Rafał Kuć
>Priority: Minor
> Fix For: 6.3
>
> Attachments: SOLR-9545.patch
>
>
> Hi, 
> Currently, when running a delta query from the Data Import Handler and pk 
> parameter is not specified Solr just logs NullPointerExeception, not 
> providing any information on what was expected. 
> Patch coming soon. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7456) PerField(DocValues|Postings)Format do not call the per-field merge methods

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7456?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510740#comment-15510740
 ] 

Michael McCandless commented on LUCENE-7456:


Wow, nice catch!  Thank you.

Our default codec postings & doc values format doesn't override the merge 
method, so I guess the impact here on a default Lucene usage is minor.  But if 
you customize your codec then it could matter if you have a better merge impl 
than the naive default.

I'll review the patch.

> PerField(DocValues|Postings)Format do not call the per-field merge methods
> --
>
> Key: LUCENE-7456
> URL: https://issues.apache.org/jira/browse/LUCENE-7456
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/codecs
>Affects Versions: 6.2.1
>Reporter: Julien MASSENET
> Attachments: LUCENE-7456.patch
>
>
> While porting some old codec code from Lucene 4.3.1, I couldn't get the 
> per-field formats to call upon the per-field merge methods; the default merge 
> method was always being called.
> I think this is a side-effect of LUCENE-5894.
> Attached is a patch with a test that reproduces the error and an associated 
> fix that pass the unit tests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7253) Make sparse doc values and segments merging more efficient

[Otis Gospodnetic (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Otis Gospodnetic updated LUCENE-7253:
-
Fix Version/s: master (7.0)

> Make sparse doc values and segments merging more efficient 
> ---
>
> Key: LUCENE-7253
> URL: https://issues.apache.org/jira/browse/LUCENE-7253
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 5.5, 6.0
>Reporter: Pawel Rog
>  Labels: performance
> Fix For: master (7.0)
>
>
> Doc Values were optimized recently to efficiently store sparse data. 
> Unfortunately there is still big problem with Doc Values merges for sparse 
> fields. When we imagine 1 billion documents index it seems it doesn't matter 
> if all documents have value for this field or there is only 1 document with 
> value. Segment merge time is the same for both cases. In most cases this is 
> not a problem but there are several cases in which one can expect having many 
> fields with sparse doc values.
> I can describe an example. During performance tests of a system with large 
> number of sparse fields I realized that Doc Values merges are a bottleneck. I 
> had hundreds of different numeric fields. Each document contained only small 
> subset of all fields. Average document contains 5-7 different numeric values. 
> As you can see data was very sparse in these fields. It turned out that 
> ingestion process was CPU-bound. Most of CPU time was spent in DocValues 
> related methods (SingletonSortedNumericDocValues#setDocument, 
> DocValuesConsumer$10$1#next, DocValuesConsumer#isSingleValued, 
> DocValuesConsumer$4$1#setNext, ...) - mostly during merging segments.
> Adrien Grand suggested to reduce the number of sparse fields and replace them 
> with smaller number of denser fields. This helped a lot but complicated 
> fields naming. 
> I am not much familiar with Doc Values source code but I have small 
> suggestion how to improve Doc Values merges for sparse fields. I realized 
> that Doc Values producers and consumers use Iterators. Let's take an example 
> of numeric Doc Values. Would it be possible to replace Iterator which 
> "travels" through all documents with Iterator over collection of non empty 
> values? Of course this would require storing object (instead of numeric) 
> which contains value and document ID. Such an iterator could significantly 
> improve merge time of sparse Doc Values fields. IMHO this won't cause big 
> overhead for dense structures but it can be game changer for sparse 
> structures.
> This is what happens in NumericDocValuesWriter on flush
> {code}
> dvConsumer.addNumericField(fieldInfo,
>new Iterable() {
>  @Override
>  public Iterator iterator() {
>return new NumericIterator(maxDoc, values, 
> docsWithField);
>  }
>});
> {code}
> Before this happens during addValue, this loop is executed to fill holes.
> {code}
> // Fill in any holes:
> for (int i = (int)pending.size(); i < docID; ++i) {
>   pending.add(MISSING);
> }
> {code}
> It turns out that variable called pending is used only internally in 
> NumericDocValuesWriter. I know pending is PackedLongValues and it wouldn't be 
> good to change it with different class (some kind of list) because this may 
> break DV performance for dense fields. I hope someone can suggest interesting 
> solutions for this problem :).
> It would be great if discussion about sparse Doc Values merge performance can 
> start here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7453) Change naming of variables/apis from docid to docnum

[Ryan Ernst (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510719#comment-15510719
 ] 

Ryan Ernst commented on LUCENE-7453:


I'm ok with {{docIndex}}, it is at least an improvement over {{docid}} 
(although I do dislike using such an overloaded term).

> Change naming of variables/apis from docid to docnum
> 
>
> Key: LUCENE-7453
> URL: https://issues.apache.org/jira/browse/LUCENE-7453
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ryan Ernst
>
> In SOLR-9528 a suggestion was made to change {{docid}} to {{docnum}}. The 
> reasoning for this is most notably that {{docid}} has a connotation about a 
> persistent unique identifier (eg like {{_id}} in elasticsearch or {{id}} in 
> solr), while {{docid}} in lucene is currently some local to a segment, and 
> not comparable directly across segments.
> When I first started working on Lucene, I had this same confusion. {{docnum}} 
> is a much better name for this transient, segment local identifier for a doc. 
> Regardless of what solr wants to do in their api (eg keeping _docid_), I 
> think we should switch the lucene apis and variable names to use docnum.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Yury Kartsev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510708#comment-15510708
 ] 

Yury Kartsev commented on SOLR-9493:


Thank you. Then definitely it's better to generate it on the client side before 
indexing. Interesting, coz now I don't see any point at all in automatically 
generating a uniqueKey field in SOLR... unless it's a single instance/1 shard 
and nobody cares about ID on the client side.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is used, the following happens (found through 
> debugging):
> Using ZK and some logic, URL list of replicas is constructed that looks like 
> this:{code}[http://replica_1_host:port/solr/collection_name/,
>  http://replica_2_host:port/solr/collection_name/,
>  http://replica_3_host:port/solr/collection_name/]{code}
> This code 

[jira] [Commented] (SOLR-9541) Support configurable authentication mechanism for internode communication

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510689#comment-15510689
 ] 

Noble Paul commented on SOLR-9541:
--

PKIAuthenticationPlugin is cheap. But not having it initialized can show up 
other errors in the rest API. The public-key is supposed to be always 
unsecured. There is no vulnerability in exposing your public key

> Support configurable authentication mechanism for internode communication
> -
>
> Key: SOLR-9541
> URL: https://issues.apache.org/jira/browse/SOLR-9541
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 5.3, 6.0
>Reporter: Hrishikesh Gadre
>
> SOLR-7849 introduced PKI based authentication mechanism for internode 
> communication. The main reason for introducing SOLR-7849 was,
> >> Relying on every Authentication plugin to secure the internode 
> >> communication is error prone. 
> At Cloudera we are using Kerberos protocol for all communication without any 
> issues (i.e. between client/server as well as server/server). We should make 
> this internode authentication mechanism configurable (with default as PKI 
> based mechanism). This will allow users to decide the appropriate 
> authentication mechanism based on their security requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9200) Add Delegation Token Support to Solr

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510668#comment-15510668
 ] 

Ishan Chattopadhyaya commented on SOLR-9200:


{quote}
Maybe I'm misunderstanding something, but don't the internode calls already use 
PKI – that seems to always be used for internode calls with SolrCloud. I don't 
see what's different with this patch than before it.
{quote}

Internode calls use PKI authentication for any plugin that does not implement 
{{HttpClientInterceptorPlugin}}. Kerberos plugin uses that "interceptor" and 
hence deals with its own internode communication (each node has a client 
principal associated with it, specified in a jaas config file, for making 
internode calls). I think the committed patch here for delegation tokens does 
not have the internode communications using the delegation tokens. If that is 
the case, we can open another issue to add a test and fix.

> Add Delegation Token Support to Solr
> 
>
> Key: SOLR-9200
> URL: https://issues.apache.org/jira/browse/SOLR-9200
> Project: Solr
>  Issue Type: New Feature
>  Components: security
>Reporter: Gregory Chanan
>Assignee: Gregory Chanan
> Fix For: 6.2, master (7.0)
>
> Attachments: SOLR-9200.patch, SOLR-9200.patch, SOLR-9200.patch, 
> SOLR-9200.patch, SOLR-9200.patch, SOLR-9200_branch_6x.patch, 
> SOLR-9200_branch_6x.patch, SOLR-9200_branch_6x.patch
>
>
> SOLR-7468 added support for kerberos authentication via the hadoop 
> authentication filter.  Hadoop also has support for an authentication filter 
> that supports delegation tokens, which allow authenticated users the ability 
> to grab/renew/delete a token that can be used to bypass the normal 
> authentication path for a time.  This is useful in a variety of use cases:
> 1) distributed clients (e.g. MapReduce) where each client may not have access 
> to the user's kerberos credentials.  Instead, the job runner can grab a 
> delegation token and use that during task execution.
> 2) If the load on the kerberos server is too high, delegation tokens can 
> avoid hitting the kerberos server after the first request
> 3) If requests/permissions need to be delegated to another user: the more 
> privileged user can request a delegation token that can be passed to the less 
> privileged user.
> Note to self:
> In 
> https://issues.apache.org/jira/browse/SOLR-7468?focusedCommentId=14579636=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14579636
>  I made the following comment which I need to investigate further, since I 
> don't know if anything changed in this area:
> {quote}3) I'm a little concerned with the "NoContext" code in KerberosPlugin 
> moving forward (I understand this is more a generic auth question than 
> kerberos specific). For example, in the latest version of the filter we are 
> using at Cloudera, we play around with the ServletContext in order to pass 
> information around 
> (https://github.com/cloudera/lucene-solr/blob/cdh5-4.10.3_5.4.2/solr/core/src/java/org/apache/solr/servlet/SolrHadoopAuthenticationFilter.java#L106).
>  Is there any way we can get the actual ServletContext in a plugin?{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 858 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/858/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.update.AutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([65E43CB083E561AB:DF3653C800CB8FBE]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:782)
at 
org.apache.solr.update.AutoCommitTest.testCommitWithin(AutoCommitTest.java:325)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:q=id:529=standard=0=20=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:775)
... 40 more


FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testBasicTextLogitStream

Re: GNU licenses inside licenses/*

[Dawid Weiss]

> I'm having a feeling of deja-vu. I thought this was discussed a long time 
> ago, [...]

Might have been and I forgot. To be honest, I don't really care, but I
thought it's an oversight.

> what we keep in ./licenses should be the *entire* LICENSE file from each
> dependency, verbatim -- regardless of if/when it's offered under multiple
> licenses -- so that if someone says "XYZ is GPL." we can say "no it's not,
> it's dual licenses CDDL or GPL, here's the entire copy of the LICENSE file
> provided with that code"

Ok, makes sense. Although to me, as a non-lawyer, by using a library
and redistributing it you're effectively agreeing to accept one of the
licenses anyway, with the freedom to pick any of the licenses offered.
And you should make it explicit which license it is. It looks odd to
me that ASF software ships with subcomponents offered under GPL... But
then, who looks at those files anyway... Let's leave it as is if Chris
says there was a discussion about it in the past.

D.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7407) Explore switching doc values to an iterator API

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510660#comment-15510660
 ] 

Michael McCandless commented on LUCENE-7407:


I opened LUCENE-7457 for the (important!) follow-on.

> Explore switching doc values to an iterator API
> ---
>
> Key: LUCENE-7407
> URL: https://issues.apache.org/jira/browse/LUCENE-7407
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>  Labels: docValues
> Fix For: master (7.0)
>
> Attachments: LUCENE-7407.patch
>
>
> I think it could be compelling if we restricted doc values to use an
> iterator API at read time, instead of the more general random access
> API we have today:
>   * It would make doc values disk usage more of a "you pay for what
> what you actually use", like postings, which is a compelling
> reduction for sparse usage.
>   * I think codecs could compress better and maybe speed up decoding
> of doc values, even in the non-sparse case, since the read-time
> API is more restrictive "forward only" instead of random access.
>   * We could remove {{getDocsWithField}} entirely, since that's
> implicit in the iteration, and the awkward "return 0 if the
> document didn't have this field" would go away.
>   * We can remove the annoying thread locals we must make today in
> {{CodecReader}}, and close the trappy "I accidentally shared a
> single XXXDocValues instance across threads", since an iterator is
> inherently "use once".
>   * We could maybe leverage the numerous optimizations we've done for
> postings over time, since the two problems ("iterate over doc ids
> and store something interesting for each") are very similar.
> This idea has come up many in the past, e.g. LUCENE-7253 is a recent
> example, and very early iterations of doc values started with exactly
> this ;)
> However, it's a truly enormous change, likely 7.0 only.  Or maybe we
> could have the new iterator APIs also ported to 6.x side by side with
> the deprecate existing random-access APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7457) Default doc values format should optimize for iterator access

[Michael McCandless (JIRA)]

Michael McCandless created LUCENE-7457:
--

 Summary: Default doc values format should optimize for iterator 
access
 Key: LUCENE-7457
 URL: https://issues.apache.org/jira/browse/LUCENE-7457
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Michael McCandless
Priority: Blocker
 Fix For: master (7.0)


In LUCENE-7407 we switched doc values consumption from random access API to an 
iterator API, but nothing was done there to improve the codec.  We should do 
that here.

At a bare minimum we should fix the existing very-sparse case to be a true 
iterator, and not wrapped with the silly legacy wrappers.

I think we should also increase the threshold (currently 1%?) when we switch 
from dense to sparse encoding.  This should fix LUCENE-7253, making merging of 
sparse doc values efficient ("pay for what you use").

I'm sure there are many other things to explore to let codecs "take advantage" 
of the fact that they no longer need to offer random access to doc values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9541) Support configurable authentication mechanism for internode communication

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510649#comment-15510649
 ] 

Hrishikesh Gadre commented on SOLR-9541:


Thanks for the feedback [~noble.paul], [~ichattopadhyaya] !

bq. AFAIK, with the kerberos plugin enabled, all internode communication is 
done via Kerberos. Every solr node has a server principal and a client 
principal. To have it use PKI, we might need to add the support.

No that is not really needed. Just having support for kerberos in all cases 
(client/server and server/server) is sufficient. 

bq. What do you mean? The documentation says it clearly

Sorry I missed that documentation section because it is in the of basic 
authentication page which I didn't go through (mostly because I am interested 
in kerberos integration and have no plans to use the basic auth). [~ctargett] 
May be we can add this information in the following page for better visibility?

https://cwiki.apache.org/confluence/display/solr/Authentication+and+Authorization+Plugins
 

I also reviewed the code again and now I think I understand the design better. 
BTW it looks like we initialize the PKIAuthenticationPlugin by default even 
when it is not used. Can we initialize PKIAuthenticationPlugin lazily (on-need 
basis) ? This will help us to avoid exposing an unsecured endpoint (to retrieve 
public-key) in case PKIAuthenticationPlugin is unused.

Any thoughts?

> Support configurable authentication mechanism for internode communication
> -
>
> Key: SOLR-9541
> URL: https://issues.apache.org/jira/browse/SOLR-9541
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 5.3, 6.0
>Reporter: Hrishikesh Gadre
>
> SOLR-7849 introduced PKI based authentication mechanism for internode 
> communication. The main reason for introducing SOLR-7849 was,
> >> Relying on every Authentication plugin to secure the internode 
> >> communication is error prone. 
> At Cloudera we are using Kerberos protocol for all communication without any 
> issues (i.e. between client/server as well as server/server). We should make 
> this internode authentication mechanism configurable (with default as PKI 
> based mechanism). This will allow users to decide the appropriate 
> authentication mechanism based on their security requirements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9446) Leader failure after creating a freshly replicated index can send nodes into recovery even if index was not changed

[Pushkar Raste (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9446?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pushkar Raste updated SOLR-9446:

Attachment: SOLR-9446.patch

> Leader failure after creating a freshly replicated index can send nodes into 
> recovery even if index was not changed
> ---
>
> Key: SOLR-9446
> URL: https://issues.apache.org/jira/browse/SOLR-9446
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: Pushkar Raste
>Assignee: Noble Paul
>Priority: Minor
> Attachments: SOLR-9446.patch
>
>
>  We noticed this issue while migrating solr index from machines {{A1, A2 and 
> A3}} to {{B1, B2, B3}}. We followed following steps (and there were no 
> updates during the migration process).
> * Index had replicas on machines {{A1, A2, A3}}. Let's say {{A1}} was the 
> leader at the time
> * We added 3 more replicas {{B1, B2 and B3}}. These nodes synced with the by 
> replication. These fresh nodes do not have tlogs.
> * We shut down one of the old nodes ({{A3}}). 
> * We then shut down the leader ({{A1}})
> * New leader got elected (let's say {{A2}}) became the new leader
> * Leader asked all the replicas to sync with it
> * Fresh nodes (ones without tlogs), first tried PeerSync but since there was 
> no frame of reference, PeerSync failed and fresh nodes fail back on to try 
> replication 
> Although replication would not copy all the segments again, it seems like we 
> can short circuit sync to put nodes back in active state as soon as possible. 
> If in case freshly replicated index becomes leader for some reason, it can 
> still send nodes (both other freshly replicated indexes and old replicas) 
> into recovery. Here is the scenario
> * Freshly replicated becomes the leader.
> * New leader however asks all the replicas to sync with it.
> * Replicas (including old one) ask for versions from the leader, but the 
> leader has no update logs, hence replicas can not compute missing versions 
> and falls back to replication



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7407) Explore switching doc values to an iterator API

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510634#comment-15510634
 ] 

Michael McCandless commented on LUCENE-7407:


bq. BTW one way that this commit could have been less massive it to split out 
the "throws IOException" additions as one change

That's a good point :)

> Explore switching doc values to an iterator API
> ---
>
> Key: LUCENE-7407
> URL: https://issues.apache.org/jira/browse/LUCENE-7407
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>  Labels: docValues
> Fix For: master (7.0)
>
> Attachments: LUCENE-7407.patch
>
>
> I think it could be compelling if we restricted doc values to use an
> iterator API at read time, instead of the more general random access
> API we have today:
>   * It would make doc values disk usage more of a "you pay for what
> what you actually use", like postings, which is a compelling
> reduction for sparse usage.
>   * I think codecs could compress better and maybe speed up decoding
> of doc values, even in the non-sparse case, since the read-time
> API is more restrictive "forward only" instead of random access.
>   * We could remove {{getDocsWithField}} entirely, since that's
> implicit in the iteration, and the awkward "return 0 if the
> document didn't have this field" would go away.
>   * We can remove the annoying thread locals we must make today in
> {{CodecReader}}, and close the trappy "I accidentally shared a
> single XXXDocValues instance across threads", since an iterator is
> inherently "use once".
>   * We could maybe leverage the numerous optimizations we've done for
> postings over time, since the two problems ("iterate over doc ids
> and store something interesting for each") are very similar.
> This idea has come up many in the past, e.g. LUCENE-7253 is a recent
> example, and very early iterations of doc values started with exactly
> this ;)
> However, it's a truly enormous change, likely 7.0 only.  Or maybe we
> could have the new iterator APIs also ported to 6.x side by side with
> the deprecate existing random-access APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9542) Kerberos delegation tokens requires missing Jackson library

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510624#comment-15510624
 ] 

Ishan Chattopadhyaya commented on SOLR-9542:


Noble, indeed lame that we have to add the old jackson library as a dependency, 
just because hadoop is stuck with an old version. Btw, I think we already have 
jackson (from org.fasterxml.*) in core.

Noble, Do you suggest we instruct users to download the jars themselves and add 
somehow them to the solr/solr.in.sh script for startup? Btw, not sure if 
upgrading Hadoop to use the latest jackson packages is an immediate option; I 
think not. [~gchanan], any thoughts? I am even fine adding it to solr-core; the 
overhead of adding this is around 750kb.

> Kerberos delegation tokens requires missing Jackson library
> ---
>
> Key: SOLR-9542
> URL: https://issues.apache.org/jira/browse/SOLR-9542
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
> Attachments: SOLR-9542.patch
>
>
> GET, RENEW or CANCEL operations for the delegation tokens support requires 
> the Solr server to have old jackson added as a dependency.
> Steps to reproduce the problem:
> 1) Configure Solr to use delegation tokens
> 2) Start Solr
> 3) Use a SolrJ application to get a delegation token.
> The server throws the following:
> {code}
> java.lang.NoClassDefFoundError: org/codehaus/jackson/map/ObjectMapper
> at 
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.managementOperation(DelegationTokenAuthenticationHandler.java:279)
> at 
> org.apache.solr.security.KerberosPlugin$RequestContinuesRecorderAuthenticationHandler.managementOperation(KerberosPlugin.java:566)
> at 
> org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:514)
> at 
> org.apache.solr.security.DelegationTokenKerberosFilter.doFilter(DelegationTokenKerberosFilter.java:123)
> at 
> org.apache.solr.security.KerberosPlugin.doAuthenticate(KerberosPlugin.java:265)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.authenticateRequest(SolrDispatchFilter.java:318)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)
> at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:208)
> at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1668)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:581)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
> at 
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1160)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:511)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1092)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
> at org.eclipse.jetty.server.Server.handle(Server.java:518)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:308)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:244)
> at 
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
> at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
> at 
> org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
> at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceAndRun(ExecuteProduceConsume.java:246)
> at 
> org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:156)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:654)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:572)
> at java.lang.Thread.run(Thread.java:745)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: GNU licenses inside licenses/*

[Chris Hostetter]

I'm having a feeling of deja-vu.

I thought this was discussed a long time ago, and the concensus was that 
what we keep in ./licenses should be the *entire* LICENSE file from each 
dependency, verbatim -- regardless of if/when it's offered under multiple 
licenses -- so that if someone says "XYZ is GPL." we can say "no it's not, 
it's dual licenses CDDL or GPL, here's the entire copy of the LICENSE file 
provided with that code"

?

: Date: Wed, 21 Sep 2016 13:50:41 +0200
: From: Dawid Weiss 
: Reply-To: dev@lucene.apache.org
: To: "dev@lucene.apache.org" 
: Subject: GNU licenses inside licenses/*
: 
: Hmm... Just peeking through licenses and noticed that there are
: multiple GNU licenses inside. These result from combined CDDL-or-GNU
: licensing options in libraries like servlet-api or mail, but perhaps
: we should truncate license files to just the CDDL part?
: 
: Dawid
: 
: -
: To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
: For additional commands, e-mail: dev-h...@lucene.apache.org
: 
: 

-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-5563) Tone down some SolrCloud logging

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated SOLR-5563:

Attachment: SOLR-5563.patch

This moves a bunch of logging from ZkController, SolrZkClient, ZkStateReader 
and the various Overseer classes from info to debug.

> Tone down some SolrCloud logging
> 
>
> Key: SOLR-5563
> URL: https://issues.apache.org/jira/browse/SOLR-5563
> Project: Solr
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Minor
> Attachments: SOLR-5563.patch, SOLR-5563.patch
>
>
> We output a *lot* of logging data from various bits of SolrCloud, a lot of 
> which is basically implementation detail that isn't particularly helpful.  
> Here's a patch to move a bunch of stuff to #debug level, rather than #info.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510499#comment-15510499
 ] 

Shawn Heisey edited comment on SOLR-9493 at 9/21/16 4:53 PM:
-

bq. Are you sure about "has no connection to the other data in the indexed 
document at all"?

Yes, I am sure.  This one line is the entirety of the code in 
UUIDUpdateProcessorFactory that sets the default value for the field:

{code:java}
  return UUID.randomUUID().toString().toLowerCase(Locale.ROOT);
{code}

This is the function being used there:

https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html#randomUUID--

Although generating an already-used ID would be rare just because of the sheer 
size of the number (128 bits), there's nothing in the code to prevent it.


was (Author: elyograg):
bq. return UUID.randomUUID().toString().toLowerCase(Locale.ROOT);

Yes, I am sure.  This one line is the entirety of the code in 
UUIDUpdateProcessorFactory that sets the default value for the field:

{code:java}
  return UUID.randomUUID().toString().toLowerCase(Locale.ROOT);
{code}

This is the function being used there:

https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html#randomUUID--

Although generating an already-used ID would be rare just because of the sheer 
size of the number (128 bits), there's nothing in the code to prevent it.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update 

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510499#comment-15510499
 ] 

Shawn Heisey commented on SOLR-9493:


bq. return UUID.randomUUID().toString().toLowerCase(Locale.ROOT);

Yes, I am sure.  This one line is the entirety of the code in 
UUIDUpdateProcessorFactory that sets the default value for the field:

{code:java}
  return UUID.randomUUID().toString().toLowerCase(Locale.ROOT);
{code}

This is the function being used there:

https://docs.oracle.com/javase/8/docs/api/java/util/UUID.html#randomUUID--

Although generating an already-used ID would be rare just because of the sheer 
size of the number (128 bits), there's nothing in the code to prevent it.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is used, the following happens (found through 
> 

[JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+136) - Build # 17867 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17867/
Java: 32bit/jdk-9-ea+136 -client -XX:+UseSerialGC

3 tests failed.
FAILED:  
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testBasicTextLogitStream

Error Message:
expected:<8.45593157553377E-6> but was:<1.0>

Stack Trace:
java.lang.AssertionError: expected:<8.45593157553377E-6> but was:<1.0>
at 
__randomizedtesting.SeedInfo.seed([F7FE8B03B9C595CE:40698D7C20B465E6]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:443)
at org.junit.Assert.assertEquals(Assert.java:512)
at 
org.apache.solr.client.solrj.io.stream.StreamExpressionTest.testBasicTextLogitStream(StreamExpressionTest.java:3133)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:535)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Yury Kartsev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510395#comment-15510395
 ] 

Yury Kartsev commented on SOLR-9493:


{quote}Solr can generate a UUID value, but it's essentially just a random 
number, and each value has no connection to the other data in the indexed 
document at all. {quote}
That's what I thought is false. That was the whole reason of why I wanted SOLR 
to generate it - to avoid that rare case when UUID matches the existing one. I 
thought SOLR uses some kind of algorithm that somehow eliminates such a case. I 
did not want to generate it on client side solely because of that reason - 
being afraid that one day it will generate an existing one. But if you're 
saying that that's what SOLR may do, then it make no difference form this point 
of view... Are you sure about "has no connection to the other data in the 
indexed document at all"? I.e. doesn't it have "counter" or "sequence-like" 
part in UUID generation algorithm?

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is 

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Yury Kartsev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510369#comment-15510369
 ] 

Yury Kartsev commented on SOLR-9493:


{quote}SolrJ is designed to send a doc to the right leader by hashing the 
id.{quote}
Very interesting... Maybe I really missed a point. Just read more about it - I 
really missed this chapter while was reading SOLR In Action book. Thanks for 
pointing it out. Although in this case the whole ability of SOLR to generate 
the uniqueKey now sounds surprising just because of what you said...

Luckily currently I have only one shard (with 3 replicas) - that's what was 
figured out to be the best for our case. But it's a very good point to consider 
in the future when we have more than one shard. Thank you.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is 

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Yury Kartsev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15510370#comment-15510370
 ] 

Yury Kartsev commented on SOLR-9493:


{quote}SolrJ is designed to send a doc to the right leader by hashing the 
id.{quote}
Very interesting... Maybe I really missed a point. Just read more about it - I 
really missed this chapter while was reading SOLR In Action book. Thanks for 
pointing it out. Although in this case the whole ability of SOLR to generate 
the uniqueKey now sounds surprising just because of what you said...

Luckily currently I have only one shard (with 3 replicas) - that's what was 
figured out to be the best for our case. But it's a very good point to consider 
in the future when we have more than one shard. Thank you.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is 

[jira] [Issue Comment Deleted] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Yury Kartsev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yury Kartsev updated SOLR-9493:
---
Comment: was deleted

(was: {quote}SolrJ is designed to send a doc to the right leader by hashing the 
id.{quote}
Very interesting... Maybe I really missed a point. Just read more about it - I 
really missed this chapter while was reading SOLR In Action book. Thanks for 
pointing it out. Although in this case the whole ability of SOLR to generate 
the uniqueKey now sounds surprising just because of what you said...

Luckily currently I have only one shard (with 3 replicas) - that's what was 
figured out to be the best for our case. But it's a very good point to consider 
in the future when we have more than one shard. Thank you.)

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is used, the