[JENKINS] Lucene-Solr-http2-Linux (64bit/jdk-11) - Build # 10 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/10/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:33055_solr, 
127.0.0.1:34979_solr, 127.0.0.1:37969_solr] Last available state: 
DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/12)={
   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node3":{   "core":"raceDeleteReplica_false_shard1_replica_n1",
   "base_url":"https://127.0.0.1:40437/solr;,   
"node_name":"127.0.0.1:40437_solr",   "state":"down",   
"type":"NRT",   "force_set_state":"false",   "leader":"true"},  
   "core_node6":{   
"core":"raceDeleteReplica_false_shard1_replica_n5",   
"base_url":"https://127.0.0.1:40437/solr;,   
"node_name":"127.0.0.1:40437_solr",   "state":"down",   
"type":"NRT",   "force_set_state":"false",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:33055_solr, 127.0.0.1:34979_solr, 127.0.0.1:37969_solr]
Last available state: 
DocCollection(raceDeleteReplica_false//collections/raceDeleteReplica_false/state.json/12)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node3":{
  "core":"raceDeleteReplica_false_shard1_replica_n1",
  "base_url":"https://127.0.0.1:40437/solr;,
  "node_name":"127.0.0.1:40437_solr",
  "state":"down",
  "type":"NRT",
  "force_set_state":"false",
  "leader":"true"},
"core_node6":{
  "core":"raceDeleteReplica_false_shard1_replica_n5",
  "base_url":"https://127.0.0.1:40437/solr;,
  "node_name":"127.0.0.1:40437_solr",
  "state":"down",
  "type":"NRT",
  "force_set_state":"false",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
at 
__randomizedtesting.SeedInfo.seed([CD1F934C10E92A7A:A709F29C781B60B0]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:224)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-11) - Build # 23274 - Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23274/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseG1GC

40 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamingTest

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([C75C6C7BE0029F9B]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.io.stream.StreamingTest

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([C75C6C7BE0029F9B]:0)


FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at http://127.0.0.1:33977/solr/collection1_shard2_replica_n3: 
Expected mime type application/octet-stream but got text/html.   
 
Error 404 Can not find: 
/solr/collection1_shard2_replica_n3/update  HTTP ERROR 
404 Problem accessing /solr/collection1_shard2_replica_n3/update. 
Reason: Can not find: 
/solr/collection1_shard2_replica_n3/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at http://127.0.0.1:33977/solr/collection1_shard2_replica_n3: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n3/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n3/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([C75C6C7BE0029F9B:3F9A954E731B0753]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

[GitHub] lucene-solr pull request #455: SOLR-12638

2018-11-28 Thread moshebla
Github user moshebla commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/455#discussion_r237364758
  
--- Diff: 
solr/core/src/test/org/apache/solr/cloud/AtomicUpdateBlockShardedTest.java ---
@@ -130,7 +130,7 @@ public void doNestedInplaceUpdateTest() throws 
Exception {
 // for now,  we know how ranges will be distributed to shards.
 // may have to look it up in clusterstate if that assumption changes.
 
-SolrInputDocument doc = sdoc("id", "a", "level_s", "root", "_root_", 
"a");
+SolrInputDocument doc = sdoc("id", "a", "level_s", "root");
--- End diff --

@dsmiley, this is what I meant by "remove hard coded _root_.
I added these since my logic depended on chidless documents having _root_.
When SOLR-5211 was merged, I just rebased on master and removed these 
hardcoded _root_ fields, which are now automatically generated by Solr.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-Tests-http2 - Build # 2 - Still Failing

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-Tests-http2/2/

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue
 {#2}

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([AB61F29AA75B604F]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([AB61F29AA75B604F]:0)




Build Log:
[...truncated 15674 lines...]
   [junit4] Suite: 
org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
   [junit4]   2> Creating dataDir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-http2/solr/build/solr-core/test/J1/temp/solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue_AB61F29AA75B604F-001/init-core-data-001
   [junit4]   2> 2255272 INFO  
(SUITE-TestSimGenericDistributedQueue-seed#[AB61F29AA75B604F]-worker) [] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 2255274 INFO  
(SUITE-TestSimGenericDistributedQueue-seed#[AB61F29AA75B604F]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 2255274 INFO  
(SUITE-TestSimGenericDistributedQueue-seed#[AB61F29AA75B604F]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> 2255275 INFO  
(TEST-TestSimGenericDistributedQueue.testDistributedQueueBlocking-seed#[AB61F29AA75B604F])
 [] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueueBlocking
   [junit4]   2> 2257377 INFO  
(TEST-TestSimGenericDistributedQueue.testDistributedQueueBlocking-seed#[AB61F29AA75B604F])
 [] o.a.s.SolrTestCaseJ4 ###Ending testDistributedQueueBlocking
   [junit4] IGNOR/A 0.00s J1 | 
TestSimGenericDistributedQueue.testDistributedQueue
   [junit4]> Assumption #1: 'badapple' test group is disabled 
(@BadApple(bugUrl=https://issues.apache.org/jira/browse/SOLR-12028))
   [junit4]   2> 2257381 INFO  
(TEST-TestSimGenericDistributedQueue.testLocallyOffer-seed#[AB61F29AA75B604F]) 
[] o.a.s.SolrTestCaseJ4 ###Starting testLocallyOffer
   [junit4]   2> 2257506 INFO  
(TEST-TestSimGenericDistributedQueue.testLocallyOffer-seed#[AB61F29AA75B604F]) 
[] o.a.s.SolrTestCaseJ4 ###Ending testLocallyOffer
   [junit4]   2> 2257509 INFO  
(TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[AB61F29AA75B604F])
 [] o.a.s.SolrTestCaseJ4 ###Starting testDistributedQueue {#2}
   [junit4]   2> Nov 29, 2018 1:52:56 PM 
com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
   [junit4]   2> WARNING: Suite execution timed out: 
org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue
   [junit4]   2>  jstack at approximately timeout time 
   [junit4]   2> 
"TEST-TestSimGenericDistributedQueue.testDistributedQueue-seed#[AB61F29AA75B604F]"
 ID=31803 TIMED_WAITING on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a6a5719
   [junit4]   2>at sun.misc.Unsafe.park(Native Method)
   [junit4]   2>- timed waiting on 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@3a6a5719
   [junit4]   2>at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   [junit4]   2>at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:194)
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.sim.GenericDistributedQueue.peek(GenericDistributedQueue.java:167)
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.sim.TestSimDistributedQueue.testDistributedQueue(TestSimDistributedQueue.java:74)
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.sim.TestSimGenericDistributedQueue.testDistributedQueue(TestSimGenericDistributedQueue.java:37)
   [junit4]   2>at sun.reflect.NativeMethodAccessorImpl.invoke0(Native 
Method)
   [junit4]   2>at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   [junit4]   2>at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   [junit4]   2>at java.lang.reflect.Method.invoke(Method.java:498)
   [junit4]   2>at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
   [junit4]   2>at 

[jira] [Commented] (SOLR-13016) Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long

2018-11-28 Thread Noble Paul (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702722#comment-16702722
 ] 

Noble Paul commented on SOLR-13016:
---

This patch brings down the total compuation time to around 1 min using caching. 
There is more to be done

> Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long
> -
>
> Key: SOLR-13016
> URL: https://issues.apache.org/jira/browse/SOLR-13016
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13016.patch
>
>
> When rules have computed values such as "#EQUAL" or "#ALL" , it takes too 
> long to compute. The problem is these values are computed too many times and 
> as the no:of nodes increase it almost takes forever. These values don't 
> change and they can be cached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-13016) Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long

2018-11-28 Thread Noble Paul (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul updated SOLR-13016:
--
Attachment: SOLR-13016.patch

> Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long
> -
>
> Key: SOLR-13016
> URL: https://issues.apache.org/jira/browse/SOLR-13016
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-13016.patch
>
>
> When rules have computed values such as "#EQUAL" or "#ALL" , it takes too 
> long to compute. The problem is these values are computed too many times and 
> as the no:of nodes increase it almost takes forever. These values don't 
> change and they can be cached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Assigned] (SOLR-13016) Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long

2018-11-28 Thread Noble Paul (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-13016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul reassigned SOLR-13016:
-

Assignee: Noble Paul

> Computing suggestions when policy have "#EQUAL" or "#ALL" rules take too long
> -
>
> Key: SOLR-13016
> URL: https://issues.apache.org/jira/browse/SOLR-13016
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
>
> When rules have computed values such as "#EQUAL" or "#ALL" , it takes too 
> long to compute. The problem is these values are computed too many times and 
> as the no:of nodes increase it almost takes forever. These values don't 
> change and they can be cached



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-http2-MacOSX (64bit/jdk-9) - Build # 2 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-MacOSX/2/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseParallelGC

8 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation

Error Message:
Error starting up MiniSolrCloudCluster

Stack Trace:
java.lang.Exception: Error starting up MiniSolrCloudCluster
at __randomizedtesting.SeedInfo.seed([BD74B4F6EB6359CA]:0)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.checkForExceptions(MiniSolrCloudCluster.java:523)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.(MiniSolrCloudCluster.java:252)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.(MiniSolrCloudCluster.java:196)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.(MiniSolrCloudCluster.java:123)
at 
org.apache.solr.cloud.TestSolrCloudWithSecureImpersonation.startup(TestSolrCloudWithSecureImpersonation.java:113)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)
Suppressed: java.lang.NoClassDefFoundError: Could not initialize class 
javax.crypto.JarVerifier
at 
java.base/javax.crypto.ProviderVerifier.verify(ProviderVerifier.java:128)
at 
java.base/javax.crypto.JceSecurity.verifyProvider(JceSecurity.java:189)
at 
java.base/javax.crypto.JceSecurity.getVerificationResult(JceSecurity.java:215)
at 
java.base/javax.crypto.JceSecurity.canUseProvider(JceSecurity.java:229)
at 
java.base/javax.crypto.KeyGenerator.nextSpi(KeyGenerator.java:363)
at 
java.base/javax.crypto.KeyGenerator.(KeyGenerator.java:176)
at 
java.base/javax.crypto.KeyGenerator.getInstance(KeyGenerator.java:244)
at 
org.apache.hadoop.security.token.SecretManager.(SecretManager.java:143)
at 
org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager.(AbstractDelegationTokenSecretManager.java:104)
at 
org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager.(ZKDelegationTokenSecretManager.java:138)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager$ZKSecretManager.(DelegationTokenManager.java:95)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenManager.(DelegationTokenManager.java:116)
at 
org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticationHandler.initTokenManager(DelegationTokenAuthenticationHandler.java:148)
at 

[jira] [Created] (LUCENE-8578) Can I do a lot of analysis on one field at the time of indexing?

2018-11-28 Thread YOO JEONGIN (JIRA)
YOO JEONGIN created LUCENE-8578:
---

 Summary: Can I do a lot of analysis on one field at the time of 
indexing?
 Key: LUCENE-8578
 URL: https://issues.apache.org/jira/browse/LUCENE-8578
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: YOO JEONGIN


Hello


I have a question about index schemas.

1) Can I do various analysis on one field?
For example, you can analyze the 'title' field with multiple tokenizers, and 
merge the analysis into a single field.

2) You can collect multiple fields in one field using 'copyField' function. 
However, several fields have different data attributes (eg, category fields, 
text fields, etc.) _)
At this time, I would like to analyze each field differently.

Do you have these features in version 7.5? Is there any kind of shortcut to do 
these similar functions?

Thank you for your advice.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.6-Windows (32bit/jdk1.8.0_172) - Build # 12 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Windows/12/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseSerialGC

7 tests failed.
FAILED:  org.apache.solr.cloud.OverseerRolesTest.testOverseerRole

Error Message:
Timed out waiting for overseer state change

Stack Trace:
java.lang.AssertionError: Timed out waiting for overseer state change
at 
__randomizedtesting.SeedInfo.seed([B3B244321D248DE3:5279B9A62697BB32]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.OverseerRolesTest.waitForNewOverseer(OverseerRolesTest.java:63)
at 
org.apache.solr.cloud.OverseerRolesTest.testOverseerRole(OverseerRolesTest.java:143)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testTriggerThrottling

Error Message:
Both triggers should have fired by now

Stack Trace:
java.lang.AssertionError: Both triggers should have fired by now
at 

[JENKINS] Lucene-Solr-repro - Build # 2037 - Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-repro/2037/

[...truncated 48 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-http2/1/consoleText

[repro] Revision: 59615cb88a9b9dad6a39b7540e0c850d2f52ea90

[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=raceConditionOnDeleteAndRegisterReplica 
-Dtests.seed=AAFD898163A334B5 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=sr-Latn-ME -Dtests.timezone=Asia/Dili -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=deleteReplicaByCountForAllShards -Dtests.seed=AAFD898163A334B5 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sr-Latn-ME 
-Dtests.timezone=Asia/Dili -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=deleteLiveReplicaTest -Dtests.seed=AAFD898163A334B5 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sr-Latn-ME 
-Dtests.timezone=Asia/Dili -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
81c092d8262a68dfda3994e790f2e1f3fdf275e2
[repro] git fetch
[repro] git checkout 59615cb88a9b9dad6a39b7540e0c850d2f52ea90

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   DeleteReplicaTest
[repro] ant compile-test

[...truncated 3674 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.DeleteReplicaTest" -Dtests.showOutput=onerror  
-Dtests.seed=AAFD898163A334B5 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=sr-Latn-ME -Dtests.timezone=Asia/Dili -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 14489 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   2/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro] git checkout 81c092d8262a68dfda3994e790f2e1f3fdf275e2

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-http2-Windows (64bit/jdk-12-ea+12) - Build # 5 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Windows/5/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:52457_solr, 
127.0.0.1:52458_solr, 127.0.0.1:52463_solr] Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/11)={
   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node3":{   "core":"raceDeleteReplica_true_shard1_replica_n1", 
  "base_url":"http://127.0.0.1:52466/solr;,   
"node_name":"127.0.0.1:52466_solr",   "state":"down",   
"type":"NRT",   "leader":"true"}, "core_node6":{   
"core":"raceDeleteReplica_true_shard1_replica_n5",   
"base_url":"http://127.0.0.1:52466/solr;,   
"node_name":"127.0.0.1:52466_solr",   "state":"down",   
"type":"NRT"}, "core_node4":{   
"core":"raceDeleteReplica_true_shard1_replica_n2",   
"base_url":"http://127.0.0.1:52457/solr;,   
"node_name":"127.0.0.1:52457_solr",   "state":"down",   
"type":"NRT",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",  
 "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:52457_solr, 127.0.0.1:52458_solr, 127.0.0.1:52463_solr]
Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/11)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node3":{
  "core":"raceDeleteReplica_true_shard1_replica_n1",
  "base_url":"http://127.0.0.1:52466/solr;,
  "node_name":"127.0.0.1:52466_solr",
  "state":"down",
  "type":"NRT",
  "leader":"true"},
"core_node6":{
  "core":"raceDeleteReplica_true_shard1_replica_n5",
  "base_url":"http://127.0.0.1:52466/solr;,
  "node_name":"127.0.0.1:52466_solr",
  "state":"down",
  "type":"NRT"},
"core_node4":{
  "core":"raceDeleteReplica_true_shard1_replica_n2",
  "base_url":"http://127.0.0.1:52457/solr;,
  "node_name":"127.0.0.1:52457_solr",
  "state":"down",
  "type":"NRT",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
at 
__randomizedtesting.SeedInfo.seed([6F2E421AB32EFFF1:53823CADBDCB53B]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:223)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

[JENKINS] Lucene-Solr-7.x-Windows (64bit/jdk-11) - Build # 903 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/903/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

8 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testTriggerThrottling

Error Message:
Both triggers should have fired by now

Stack Trace:
java.lang.AssertionError: Both triggers should have fired by now
at 
__randomizedtesting.SeedInfo.seed([F6E2694D56A2CC6A:DC0C16884082FF8]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testTriggerThrottling(TriggerIntegrationTest.java:270)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)


FAILED:  
org.apache.solr.cloud.autoscaling.TriggerIntegrationTest.testTriggerThrottling

Error Message:
Both triggers should have fired by now

Stack Trace:
java.lang.AssertionError: Both triggers should have fired by now
at 
__randomizedtesting.SeedInfo.seed([F6E2694D56A2CC6A:DC0C16884082FF8]:0)
at 

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_172) - Build # 3155 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3155/
Java: 64bit/jdk1.8.0_172 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at http://127.0.0.1:42857/solr/collection1_shard2_replica_n2: 
Expected mime type application/octet-stream but got text/html.   
 
Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update  HTTP ERROR 
404 Problem accessing /solr/collection1_shard2_replica_n2/update. 
Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at http://127.0.0.1:42857/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([CEE8185D12A9F822:362EE16881B060EA]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 

[JENKINS] Lucene-Solr-repro - Build # 2035 - Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-repro/2035/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/389/consoleText

[repro] Revision: 9dd1949749916c36f8b62557f6d3b277f589f00c

[repro] Ant options: -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  -Dtestcase=HdfsRestartWhileUpdatingTest 
-Dtests.method=test -Dtests.seed=64E46779FF6ECF5F -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=es-BO -Dtests.timezone=SystemV/PST8PDT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=HdfsRestartWhileUpdatingTest 
-Dtests.seed=64E46779FF6ECF5F -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=es-BO -Dtests.timezone=SystemV/PST8PDT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
81c092d8262a68dfda3994e790f2e1f3fdf275e2
[repro] git fetch
[repro] git checkout 9dd1949749916c36f8b62557f6d3b277f589f00c

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   HdfsRestartWhileUpdatingTest
[repro] ant compile-test

[...truncated 3586 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.HdfsRestartWhileUpdatingTest" -Dtests.showOutput=onerror 
-Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.seed=64E46779FF6ECF5F -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.x/test-data/enwiki.random.lines.txt
 -Dtests.locale=es-BO -Dtests.timezone=SystemV/PST8PDT -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 59754 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: org.apache.solr.cloud.hdfs.HdfsRestartWhileUpdatingTest
[repro] git checkout 81c092d8262a68dfda3994e790f2e1f3fdf275e2

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 222 - Still Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/222/

2 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:42702/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:42702/collection1
at 
__randomizedtesting.SeedInfo.seed([386DA1A93D96242D:B0399E73936A49D5]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:509)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:498)
at 
org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

RE: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Uwe Schindler
How about just enabling ALPN and HTTP2+TLS in Java 9 or later. In that case we 
would ship with the official Java 9 libraries of Jetty, but don’t enable 
HTTP/2+TLS on Java 8? Should be easy to detect on startup.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de

 

From: Đạt Cao Mạnh  
Sent: Wednesday, November 28, 2018 5:27 PM
To: Solr/Lucene Dev 
Subject: Re: Poll: Merge jira/http2 to master branch

 

Hi Uwe,

 

To enable Conscrypt we have include several related Jetty libraries, the same 
thing will need to be done if we uses Java 9 ALPN. So support both security 
providers and testing them is kinda painful in my point of view.

 

 

On Wed, Nov 28, 2018 at 4:16 PM Đạt Cao Mạnh mailto:caomanhdat...@gmail.com> > wrote:

Hi folks, Uwe

 

After upgrading to Conscrypt 1.4.1. I still failure on 
https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/7/consoleText. I kinda 
feel that this SSL is big pain but can be solved easily in Java 9. Should it is 
possible for us to not supporting SSL in HTTP2 (anyone want to run SSL can use 
any HTTP2 proxy like nginx as a gate keeper). 

 

Thanks!

 

On Wed, Nov 28, 2018 at 10:55 AM Uwe Schindler mailto:u...@thetaphi.de> > wrote:

Hi,

 

About the bundling of conscrypt.jar in the binary distribution of Apache Solr, 
I opened LEGAL-425:

https://issues.apache.org/jira/browse/LEGAL-425

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de  

 

From: Uwe Schindler mailto:u...@thetaphi.de> > 
Sent: Wednesday, November 28, 2018 11:38 AM
To: dev@lucene.apache.org  
Subject: RE: Poll: Merge jira/http2 to master branch

 

Hi Dat, hi Shawn,

 

Thanks for the confirmation! This is what I had in mind (missing ALPN support).

 

IMHO, we should not yet switch away from Java 8 as minimum requirement. With 
multi-release JARs we are at the moment perfectly able to handle users with 
newer Java versions, but we should still support Java 8 (as its still widely 
deployed), although EOL is close. As Redhat, AdoptOpenJDK, and several Linux 
distributions still provide support for Java 8 at least for 2 or 3 more years, 
I think it’s to early to switch. With recent changes, the support by Oracle is 
no longer the way to go, as there are many alternatives.

 

My proposal would be to wait until master is branched to “branch_8x” (stays on 
Java 8 + MR-JAR support), then change “master” to have Java 11 as minimum 
requirement.

 

About HTTP2: I have no problem with bundling conscrypt (is it needed for both 
Jetty Server and Jetty Client)?, but disable it by default and only register it 
in the Java TLS SPI, if Java 8 is used. On Java 9 and later use the shipped 
HTTP2 support. My proposal would be to do it with a Multirelease JAR (this is 
currently enabled in Lucene already).

 

If there are license problems, we can do the following: Disable HTTP2 on Java 8 
completely but provide documentation in the Solr Ref Guide how to enable it 
(something like: download conscrypt-uber.jar from maven and save in solr/lib 
directory). We can enable it automatically, if class.forName() does not throw 
CNFE). Maybe add a sysprop, but that’s not needed. I think enabling conscrypt 
is easy with reflection only, or do we need to compile hard against it. From 
what I figured out, it’s only used at a few places.

 

SolrJ should by default ship without conscyrpt (make it “optional” maven 
dependency). If it’s in classpath, enable it.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de  

 

From: Đạt Cao Mạnh mailto:caomanhdat...@gmail.com> > 
Sent: Wednesday, November 28, 2018 11:01 AM
To: Solr/Lucene Dev mailto:dev@lucene.apache.org> >
Subject: Re: Poll: Merge jira/http2 to master branch

 

Hi,

 

I will try to summary all the options here

1. No support for HTTP/2 + SSL at all, if users want to run SSL they must stick 
with HTTP/1.1

2. Set Java requirement to 9 and use ALPN implementation of JDK 9

3. If users want to use HTTP/2, we will notice about license problem then 
download and use Conscrypt library. Of course that we still test the Conscrypt 
implementation in tests.

 

 

On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey mailto:apa...@elyograg.org> > wrote:

On 11/27/2018 3:32 AM, Uwe Schindler wrote:
> I'd like to bring one thing into attention: This library conscrypt is 
> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork 
> named BoringSSL. I'd recommend to check the licensing, because OpenSSL 
> licenses are a bit strange (some BSD fork, also ASF2, but some code is still 
> outdated - I am not sure what the fork actually uses). I have the feeling we 
> should include ASF legal department.
>
> Nevertheless, I am not 

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-11) - Build # 7639 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7639/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseG1GC

12 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:60171_solr, 
127.0.0.1:60172_solr, 127.0.0.1:60173_solr] Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/14)={
   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node3":{   "core":"raceDeleteReplica_true_shard1_replica_n1", 
  "base_url":"https://127.0.0.1:60174/solr;,   
"node_name":"127.0.0.1:60174_solr",   "state":"down",   
"type":"NRT",   "leader":"true"}, "core_node6":{   
"core":"raceDeleteReplica_true_shard1_replica_n5",   
"base_url":"https://127.0.0.1:60174/solr;,   
"node_name":"127.0.0.1:60174_solr",   "state":"down",   
"type":"NRT"}, "core_node4":{   
"core":"raceDeleteReplica_true_shard1_replica_n2",   
"base_url":"https://127.0.0.1:60173/solr;,   
"node_name":"127.0.0.1:60173_solr",   "state":"down",   
"type":"NRT",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",  
 "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:60171_solr, 127.0.0.1:60172_solr, 127.0.0.1:60173_solr]
Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/14)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node3":{
  "core":"raceDeleteReplica_true_shard1_replica_n1",
  "base_url":"https://127.0.0.1:60174/solr;,
  "node_name":"127.0.0.1:60174_solr",
  "state":"down",
  "type":"NRT",
  "leader":"true"},
"core_node6":{
  "core":"raceDeleteReplica_true_shard1_replica_n5",
  "base_url":"https://127.0.0.1:60174/solr;,
  "node_name":"127.0.0.1:60174_solr",
  "state":"down",
  "type":"NRT"},
"core_node4":{
  "core":"raceDeleteReplica_true_shard1_replica_n2",
  "base_url":"https://127.0.0.1:60173/solr;,
  "node_name":"127.0.0.1:60173_solr",
  "state":"down",
  "type":"NRT",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
at 
__randomizedtesting.SeedInfo.seed([DDA0A23578E2412C:B7B6C3E510100BE6]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:223)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

Re: Jenkins test report improvements: Suspicious (Recent) Failure Rates

2018-11-28 Thread Chris Hostetter


: first place. Some things were wrapped in "multiple-failure" exception,
: which didn't seem right to me. When I wrote the dedicated runner there
: was a lot of guessing involved, including XML reports (this bit not
: even part of JUnit, but Apache Ant and Apache Maven).
... 
: It's actually the listener that I wrote that emits this log. Which
: leads me to say that if it's more helpful we could emit a different
: type of report from the runner (in addition to what Jenkins consumes).
: Internally, listeners receive each and every exception separately so
: they can serialize them in a more palatable manner. The XML listener
: is just one implementation:

Interesting point ... but if we did generate a "new" structured report 
from the runner, something would still need to be able to aggregate those 
reports across multiple runs (ie: "ant test" being run in multiple dirs, 
the repro logic re-running "ant test" later in the same jenkins job, 
etc...) for it to be useful.  As long as we stick w/the constraints of 
the JUnit XML report we can leverage the ((ant|maven)+jenkins) testReport.

Thinking about it purely from the point of view of the failure rate 
reports i've been generating, it would be nice if *every* TestClass 
invocation consistently included a single psuedo-method entry indicating 
either a PASS if there were no suite level initialization or tear down 
problems, and a FAIL if there were -- something akin to the existing 
"junit.framework.TestSuite" psuedo test method entries, but exactly one 
per "suite instance" regardless of wether it succeeds or not (unlike the 
way there can currently be multiple "junit.framework.TestSuite" instances 
in a single class if it has multiple problems, but none if it doesn't) ... 
that way the number of runs is just the exact number of entries with this 
special name, and the number of suite instances that had 1 or more 
problems is just the number of psuedo-mehtods that "FAILed"


It would make the xml files bigger in the "no problem" situation, but not 
by much.


-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.6-Linux (32bit/jdk1.8.0_172) - Build # 39 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/39/
Java: 32bit/jdk1.8.0_172 -server -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at 
https://127.0.0.1:45099/solr/collection1_shard2_replica_n2: Expected mime type 
application/octet-stream but got text/html.Error 404 
Can not find: /solr/collection1_shard2_replica_n2/update  
HTTP ERROR 404 Problem accessing 
/solr/collection1_shard2_replica_n2/update. Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at https://127.0.0.1:45099/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([93D1DC466E605DC3:6B172573FD79C50B]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.request.UpdateRequest.commit(UpdateRequest.java:237)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned(CloudSolrClientTest.java:725)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 

[JENKINS] Lucene-Solr-7.x-Solaris (64bit/jdk1.8.0) - Build # 934 - Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Solaris/934/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: 
[127.0.0.1:10218_solr, 127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr] Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/33)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   
"autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{   
"replicas":{"core_node1":{   
"core":"testCreateCollectionAddReplica_shard1_replica_n1",   
"SEARCHER.searcher.maxDoc":0,   "SEARCHER.searcher.deletedDocs":0,  
 "INDEX.sizeInBytes":10240,   "node_name":"127.0.0.1:10217_solr",   
"state":"active",   "type":"NRT",   
"INDEX.sizeInGB":9.5367431640625E-6,   "SEARCHER.searcher.numDocs":0}}, 
  "range":"8000-7fff",   "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10218_solr, 127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr]
Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/33)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
  "replicas":{"core_node1":{
  "core":"testCreateCollectionAddReplica_shard1_replica_n1",
  "SEARCHER.searcher.maxDoc":0,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":10240,
  "node_name":"127.0.0.1:10217_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":9.5367431640625E-6,
  "SEARCHER.searcher.numDocs":0}},
  "range":"8000-7fff",
  "state":"active"}}}
at 
__randomizedtesting.SeedInfo.seed([A43B06EC9D0D64F5:241B63C28C4E8C53]:0)
at 
org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 

[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 227 - Still Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/227/

2 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLowerTerms

Error Message:
Could not load collection from ZK: collection1

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from ZK: 
collection1
at 
__randomizedtesting.SeedInfo.seed([2A6C082FBBAC0D85:A458286C7B0BADF1]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1321)
at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:737)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:148)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:131)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.getTotalReplicas(AbstractFullDistribZkTestBase.java:495)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createJettys(AbstractFullDistribZkTestBase.java:448)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createServers(AbstractFullDistribZkTestBase.java:341)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1008)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.zookeeper.KeeperException$SessionExpiredException: 
KeeperErrorCode = Session expired for /collections/collection1/state.json
at 

[JENKINS] Lucene-Solr-http2-Linux (64bit/jdk-11) - Build # 9 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/9/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at 
https://127.0.0.1:41351/solr/collection1_shard2_replica_n3: Expected mime type 
application/octet-stream but got text/html.Error 404 
Can not find: /solr/collection1_shard2_replica_n3/update  
HTTP ERROR 404 Problem accessing 
/solr/collection1_shard2_replica_n3/update. Reason: Can not find: 
/solr/collection1_shard2_replica_n3/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.14.v20181114  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at https://127.0.0.1:41351/solr/collection1_shard2_replica_n3: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n3/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n3/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n3/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.14.v20181114




at 
__randomizedtesting.SeedInfo.seed([F619394265115ECC:34AE052A6651AEB4]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-6595) Improve error response in case distributed collection cmd fails

2018-11-28 Thread Jason Gerlowski (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702292#comment-16702292
 ] 

Jason Gerlowski commented on SOLR-6595:
---

Thinking aloud here, and I guess also soliciting feedback.

The current patch sets 500 as the value for the "status' property, as well as 
the HTTP status code on the response.  The expectation in most other places 
seems to be that the "status" property matches the HTTP status code.  So this 
seems like the technically correct thing to do from an API perspective.

There's is a downside to this though- SolrJ converts non-200 responses into 
exceptions.  So while the failure information is still in the response, SolrJ 
users can't get at it.  (This isn't strictly true...SolrJ tries its best to 
come up with a good exception message by looking for properties like "error" 
and "failure".  But that's a pale substitute to giving users access to the 
response itself if they want it).

It'd be cool if SolrJ users could access the original response in exceptional 
cases.  Maybe we should attach the parsed NamedList to RemoteSolrExceptions 
that get thrown by SolrJ.  That seems like a separate JIRA, but wanted to raise 
it here since it bears on these response changes indirectly.

> Improve error response in case distributed collection cmd fails
> ---
>
> Key: SOLR-6595
> URL: https://issues.apache.org/jira/browse/SOLR-6595
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 4.10
> Environment: SolrCloud with Client SSL
>Reporter: Sindre Fiskaa
>Assignee: Jason Gerlowski
>Priority: Minor
> Attachments: SOLR-6595.patch
>
>
> Followed the description 
> https://cwiki.apache.org/confluence/display/solr/Enabling+SSL and generated a 
> self signed key pair. Configured a few solr-nodes and used the collection api 
> to crate a new collection. -I get error message when specify the nodes with 
> the createNodeSet param. When I don't use createNodeSet param the collection 
> gets created without error on random nodes. Could this be a bug related to 
> the createNodeSet param?- *Update: It failed due to what turned out to be 
> invalid client certificate on the overseer, and returned the following 
> response:*
> {code:xml}
> 
>   0 name="QTime">185
>   
> org.apache.solr.client.solrj.SolrServerException:IOException occured 
> when talking to server at: https://vt-searchln04:443/solr
>   
> 
> {code}
> *Update: Three problems:*
> # Status=0 when the cmd did not succeed (only ZK was updated, but cores not 
> created due to failing to connect to shard nodes to talk to core admin API).
> # The error printed does not tell which action failed. Would be helpful to 
> either get the msg from the original exception or at least some message 
> saying "Failed to create core, see log on Overseer 
> # State of collection is not clean since it exists as far as ZK is concerned 
> but cores not created. Thus retrying the CREATECOLLECTION cmd would fail. 
> Should Overseer detect error in distributed cmds and rollback changes already 
> made in ZK?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12209) add Paging Streaming Expression

2018-11-28 Thread Lucene/Solr QA (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702296#comment-16702296
 ] 

Lucene/Solr QA commented on SOLR-12209:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | {color:red} patch {color} | {color:red}  0m  5s{color} 
| {color:red} SOLR-12209 does not apply to master. Rebase required? Wrong 
Branch? See 
https://wiki.apache.org/solr/HowToContribute#Creating_the_patch_file for help. 
{color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-12209 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12926316/SOLR-12209.patch |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/235/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> add Paging Streaming Expression
> ---
>
> Key: SOLR-12209
> URL: https://issues.apache.org/jira/browse/SOLR-12209
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: streaming expressions
>Reporter: mosh
>Priority: Major
> Attachments: SOLR-12209.patch, SOLR-12209.patch, SOLR-12209.patch
>
>
> Currently the closest streaming expression that allows some sort of 
> pagination is top.
> I propose we add a new streaming expression, which is based on the 
> RankedStream class to add offset to the stream. currently it can only be done 
> in code by reading the stream until the desired offset is reached.
> The new expression will be used as such:
> {{paging(rows=3, search(collection1, q="*:*", qt="/export", 
> fl="id,a_s,a_i,a_f", sort="a_f desc, a_i desc"), sort="a_f asc, a_i asc", 
> start=100)}}
> {{this will offset the returned stream by 100 documents}}
>  
> [~joel.bernstein] what to you think?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8563) Remove k1+1 from the numerator of BM25Similarity

2018-11-28 Thread Luca Cavanna (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702294#comment-16702294
 ] 

Luca Cavanna commented on LUCENE-8563:
--

I opened [https://github.com/apache/lucene-solr/pull/511] . 

> Remove k1+1 from the numerator of  BM25Similarity
> -
>
> Key: LUCENE-8563
> URL: https://issues.apache.org/jira/browse/LUCENE-8563
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Our current implementation of BM25 does
> {code:java}
> boost * IDF * (k1+1) * tf / (tf + norm)
> {code}
> As (k1+1) is a constant, it is the same for every term and doesn't modify 
> ordering. It is often omitted and I found out that the "The Probabilistic 
> Relevance Framework: BM25 and Beyond" paper by Robertson (BM25's author) and 
> Zaragova even describes adding (k1+1) to the numerator as a variant whose 
> benefit is to be more comparable with Robertson/Sparck-Jones weighting, which 
> we don't care about.
> {quote}A common variant is to add a (k1 + 1) component to the
>  numerator of the saturation function. This is the same for all
>  terms, and therefore does not affect the ranking produced.
>  The reason for including it was to make the final formula
>  more compatible with the RSJ weight used on its own
> {quote}
> Should we remove it from BM25Similarity as well?
> A side-effect that I'm interested in is that integrating other score 
> contributions (eg. via oal.document.FeatureField) would be a bit easier to 
> reason about. For instance a weight of 3 in FeatureField#newSaturationQuery 
> would have a similar impact as a term whose IDF is 3 (and thus docFreq ~= 5%) 
> rather than a term whose IDF is 3/(k1 + 1).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-7414) CSVResponseWriter returns empty field when fl alias is combined with '*' selector

2018-11-28 Thread Lucene/Solr QA (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702289#comment-16702289
 ] 

Lucene/Solr QA commented on SOLR-7414:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
43s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  1m 35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  1m 35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  1m 35s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 47m 
48s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 53m 40s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-7414 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12949439/SOLR-7414.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 81c092d |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | 1.8.0_191 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/234/testReport/ |
| modules | C: solr/core U: solr/core |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/234/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> CSVResponseWriter returns empty field when fl alias is combined with '*' 
> selector
> -
>
> Key: SOLR-7414
> URL: https://issues.apache.org/jira/browse/SOLR-7414
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers
>Reporter: Michael Lawrence
>Priority: Major
> Attachments: SOLR-7414-old.patch, SOLR-7414.patch, SOLR-7414.patch
>
>
> Attempting to retrieve all fields while renaming one, e.g., "inStock" to 
> "stocked" (URL below), results in CSV output that has a column for "inStock" 
> (should be "stocked"), and the column has no values. 
> steps to reproduce using 5.1...
> {noformat}
> $ bin/solr -e techproducts
> ...
> $ curl -X POST -H 'Content-Type: application/json' 
> 'http://localhost:8983/solr/techproducts/update?commit=true' --data-binary 
> '[{ "id" : "aaa", "bar_i" : 7, "inStock" : true }, { "id" : "bbb", "bar_i" : 
> 7, "inStock" : false }, { "id" : "ccc", "bar_i" : 7, "inStock" : true }]'
> {"responseHeader":{"status":0,"QTime":730}}
> $ curl 
> 'http://localhost:8983/solr/techproducts/query?q=bar_i:7=id,stocked:inStock=csv'
> id,stocked
> aaa,true
> bbb,false
> ccc,true
> $ curl 
> 'http://localhost:8983/solr/techproducts/query?q=bar_i:7=*,stocked:inStock=csv'
> bar_i,id,_version_,inStock
> 7,aaa,1498719888088236032,
> 7,bbb,1498719888090333184,
> 7,ccc,1498719888090333185,
> $ curl 
> 'http://localhost:8983/solr/techproducts/query?q=bar_i:7=stocked:inStock,*=csv'
> bar_i,id,_version_,inStock
> 7,aaa,1498719888088236032,
> 7,bbb,1498719888090333184,
> 7,ccc,1498719888090333185,
> {noformat}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8517) TestRandomChains.testRandomChainsWithLargeStrings failure

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702264#comment-16702264
 ] 

ASF subversion and git services commented on LUCENE-8517:
-

Commit 54907903e8d1a5da0c65328f24a1018c5e393afc in lucene-solr's branch 
refs/heads/jira/http2 from Michael Sokolov
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5490790 ]

LUCENE-8517: do not wrap FixedShingleFilter with conditional in TestRandomChains


> TestRandomChains.testRandomChainsWithLargeStrings failure
> -
>
> Key: LUCENE-8517
> URL: https://issues.apache.org/jira/browse/LUCENE-8517
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Reporter: Steve Rowe
>Priority: Major
>  Time Spent: 2h 10m
>  Remaining Estimate: 0h
>
> From 
> [https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/2828/consoleText], 
> reproduces for me on Java8:
> {noformat}
> Checking out Revision 216f10026b86627750e133fe24ce6a750c470695 
> (refs/remotes/origin/branch_7x)
> [...]
> [java-info] java version "10.0.1"
> [java-info] OpenJDK Runtime Environment (10.0.1+10, Oracle Corporation)
> [java-info] OpenJDK 64-Bit Server VM (10.0.1+10, Oracle Corporation)
> [java-info] Test args: [-XX:-UseCompressedOops -XX:+UseConcMarkSweepGC]
> [...]
>[junit4] Suite: org.apache.lucene.analysis.core.TestRandomChains
>[junit4]   2> Exception from random analyzer: 
>[junit4]   2> charfilters=
>[junit4]   2>   
> org.apache.lucene.analysis.charfilter.MappingCharFilter(org.apache.lucene.analysis.charfilter.NormalizeCharMap@3ef95503,
>  java.io.StringReader@70dde633)
>[junit4]   2>   
> org.apache.lucene.analysis.fa.PersianCharFilter(org.apache.lucene.analysis.charfilter.MappingCharFilter@12423b20)
>[junit4]   2> tokenizer=
>[junit4]   2>   org.apache.lucene.analysis.th.ThaiTokenizer()
>[junit4]   2> filters=
>[junit4]   2>   
> org.apache.lucene.analysis.compound.HyphenationCompoundWordTokenFilter(ValidatingTokenFilter@7914bba7
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,
>  org.apache.lucene.analysis.compound.hyphenation.HyphenationTree@abd7bca)
>[junit4]   2>   
> Conditional:org.apache.lucene.analysis.MockGraphTokenFilter(java.util.Random@56348091,
>  OneTimeWrapper@aa1c073 
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1)
>[junit4]   2>   
> Conditional:org.apache.lucene.analysis.shingle.FixedShingleFilter(OneTimeWrapper@4cf58fce
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,
>  4, , )
>[junit4]   2>   
> org.apache.lucene.analysis.pt.PortugueseLightStemFilter(ValidatingTokenFilter@3a915324
>  
> term=,bytes=[],startOffset=0,endOffset=0,positionIncrement=1,positionLength=1,type=word,termFrequency=1,keyword=false)
>[junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestRandomChains 
> -Dtests.method=testRandomChainsWithLargeStrings -Dtests.seed=92344C536D4E00F4 
> -Dtests.multiplier=3 -Dtests.slow=true -Dtests.locale=en-ZW 
> -Dtests.timezone=Atlantic/Faroe -Dtests.asserts=true 
> -Dtests.file.encoding=US-ASCII
>[junit4] ERROR   0.46s J2 | 
> TestRandomChains.testRandomChainsWithLargeStrings <<<
>[junit4]> Throwable #1: java.lang.IllegalStateException: stage 3: 
> inconsistent startOffset at pos=0: 0 vs 5; token=effort
>[junit4]>  at 
> __randomizedtesting.SeedInfo.seed([92344C536D4E00F4:F86FF34234002007]:0)
>[junit4]>  at 
> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:109)
>[junit4]>  at 
> org.apache.lucene.analysis.pt.PortugueseLightStemFilter.incrementToken(PortugueseLightStemFilter.java:48)
>[junit4]>  at 
> org.apache.lucene.analysis.ValidatingTokenFilter.incrementToken(ValidatingTokenFilter.java:68)
>[junit4]>  at 
> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkResetException(BaseTokenStreamTestCase.java:441)
>[junit4]>  at 
> org.apache.lucene.analysis.BaseTokenStreamTestCase.checkRandomData(BaseTokenStreamTestCase.java:546)
>[junit4]>  at 
> org.apache.lucene.analysis.core.TestRandomChains.testRandomChainsWithLargeStrings(TestRandomChains.java:897)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]>  at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]>  at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>[junit4]>  at 
> 

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702265#comment-16702265
 ] 

ASF subversion and git services commented on SOLR-5211:
---

Commit 1534bbe4ae593521ba27c92495cf52c54be85fbf in lucene-solr's branch 
refs/heads/jira/http2 from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=1534bbe ]

SOLR-5211: fix test


> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12984) The search Streaming Expression should properly support and push down paging when using the /select handler

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702266#comment-16702266
 ] 

ASF subversion and git services commented on SOLR-12984:


Commit c2cac887702f9efc0a6bf75cd9f1e78f730c2c4f in lucene-solr's branch 
refs/heads/jira/http2 from [~joel.bernstein]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c2cac88 ]

SOLR-12984: The search Streaming Expression should properly support and push 
down paging when using the /select handler


> The search Streaming Expression should properly support and push down paging 
> when using the /select handler
> ---
>
> Key: SOLR-12984
> URL: https://issues.apache.org/jira/browse/SOLR-12984
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-12984.patch, SOLR-12984.patch, SOLR-12984.patch, 
> SOLR-12984.patch
>
>
> Currently the search Streaming Expression doesn't properly support paging 
> even when going to the /select handler. This is due to very old 
> implementation decisions that were geared towards supporting streaming entire 
> result sets from the /export handler. It's time to change this behavior the 
> so the search expression can be used to handle typical paging scenarios.
> This ticket will maintain the same behavior for when qt=/export, but will 
> push down 'rows' and 'start' parameters when using /select handler. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8573) BKDWriter should use FutureArrays#mismatch

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702268#comment-16702268
 ] 

ASF subversion and git services commented on LUCENE-8573:
-

Commit ecce083c5017576f552a5fd8e347b8748dbc0db9 in lucene-solr's branch 
refs/heads/jira/http2 from [~cbuescher]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ecce083 ]

LUCENE-8573: Use FutureArrays#mismatch in BKDWriter

Closes #510

Signed-off-by: Adrien Grand 


> BKDWriter should use FutureArrays#mismatch
> --
>
> Key: LUCENE-8573
> URL: https://issues.apache.org/jira/browse/LUCENE-8573
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> In a number of places, BKDWriter tries to find the first mismatching byte 
> between multiple arrays with a for loop. It could use the safer 
> FutureArrays#mismatch instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702269#comment-16702269
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit 2611c22b6b00d73f6ae1fc6996e029f73ee7458d in lucene-solr's branch 
refs/heads/jira/http2 from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2611c22 ]

SOLR-12398: retroactively add CHANGES.txt back-compat break for 7.5


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Fix For: 7.5
>
> Attachments: SOLR-12398.patch, SOLR-12398.patch, 
> repro-facet-heatmaps-response-format-change.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12497) Add ref guide docs for Hadoop Credential Provider based SSL/TLS store password source.

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702267#comment-16702267
 ] 

ASF subversion and git services commented on SOLR-12497:


Commit 2715beb6df77d7e5795b9c111a37178527cf3831 in lucene-solr's branch 
refs/heads/jira/http2 from [~ctargett]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2715beb ]

SOLR-12497: fix typo


> Add ref guide docs for Hadoop Credential Provider based SSL/TLS store 
> password source.
> --
>
> Key: SOLR-12497
> URL: https://issues.apache.org/jira/browse/SOLR-12497
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: documentation
>Affects Versions: 7.4
>Reporter: Mano Kovacs
>Assignee: Cassandra Targett
>Priority: Minor
> Fix For: 7.6, master (8.0)
>
> Attachments: SOLR-12497.patch
>
>
> Document configuration added in SOLR-10783.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8529) Use the completion key to tiebreak completion suggestion

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702270#comment-16702270
 ] 

ASF subversion and git services commented on LUCENE-8529:
-

Commit 81c092d8262a68dfda3994e790f2e1f3fdf275e2 in lucene-solr's branch 
refs/heads/jira/http2 from [~jim.ferenczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=81c092d ]

LUCENE-8529: TopSuggestDocsCollector will now use the completion key to 
tiebreak completion
suggestions with identical scores.


> Use the completion key to tiebreak completion suggestion
> 
>
> Key: LUCENE-8529
> URL: https://issues.apache.org/jira/browse/LUCENE-8529
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Fix For: master (8.0), 7.7
>
> Attachments: LUCENE-8529.patch
>
>
> Today the completion suggester uses the document id to tiebreak completion 
> suggestion with same scores. It would improve the stability of the sort to 
> use the surface form of suggestions as the first tiebreaker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 37 - Still Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/37/

2 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue

Error Message:
action wasn't interrupted

Stack Trace:
java.lang.AssertionError: action wasn't interrupted
at 
__randomizedtesting.SeedInfo.seed([BB5D3569FE6D602A:72E877C7F70AA6DF]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testEventQueue(TestSimTriggerIntegration.java:686)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at http://127.0.0.1:43566/solr/collection1_shard2_replica_n2: 
Expected mime type application/octet-stream but got text/html.   
 
Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update  

Re: SynonymQuery / Query Expansion Strategies Discussion

2018-11-28 Thread Doug Turnbull
I like that idea Alan. The trick is for QueryBuilder's 'newSynonymQuery' to
be useful in that context, you need to pass terms with metadata down to the
subclass. This is what I started working on a few weeks ago:

https://github.com/o19s/lucene-solr/commit/0fc3930671ef002cfbb5e3d52b6f8edc3715bf14

I don't think it's as simple as overriding analyzeBoolean/analyzeMultiBoolean
as Rob suggests, as there's also analyzeGraphBoolean and the  that would
also need to collect this metadata. I wouldn't want to copy paste all this
code into a subclass just to add one token attribute.

-Doug



On Wed, Nov 28, 2018 at 12:25 PM Alan Woodward  wrote:

> I think we can expose this information now with a small tweak to the
> SynonymGraphFilter, using the already-existing TypeAttribute.
>
> SGF is hard-coded to set the type attribute to “SYNONYM” on all tokens
> that it inserts into the stream.  It should be simple to add another
> constructor parameter allowing users to change this; then you can chain
> synonym filters, one for each type of expansion you want: synonym, hyponym,
> hypernym, whatever, each setting the type attribute differently.
>
> > On 28 Nov 2018, at 15:59, Michael Gibney 
> wrote:
> >
> > I think the objection to "boosting" in token filters isn't because it
> > is "too much", but rather because it breaks the abstraction of the
> > analysis chain to directly target scoring (as implied by
> > characterizing as "boosting").
> >
> > That said, I'm sympathetic to an approach that would establish an
> > Attribute to expose the kind of information that would be useful in
> > the context of synonyms (or other sorts of derived tokens discussed
> > here, where it could be useful to express information about token
> > derivation). Such an Attribute would not be directly related to
> > scoring/boosting, but would be related to analysis per se, (e.g.,
> > source token text, thesaurus, degree of confidence, etc.); support
> > could be selectively implemented by TokenFilters, and optionally
> > leveraged by query builders (e.g., translated to boosts) or even
> > recorded to index Payloads by a final custom analysis component 
> >
> > "You can look at any attribute on the tokenstream you want", "rely on
> > abstract attributes (type, ...) then it should be easy to sub-class
> > the query builder to access them".  Obviously that works iff analysis
> > components record the relevant information in attributes on the
> > tokenstream, which I think they currently don't (for much of the
> > information that has been discussed here) ... and I know of no
> > standard way to express the relevant information on the tokenstream.
> >
> > I can see that such an Attribute would be out of place (too
> > specialized) in the context of the Attributes in lucene/core; but
> > there are lots of more specialized Attributes in the various
> > submodules under lucene/analysis/* (SynonymGraphFilter lives in
> > analysis-common, FWIW). Again, this doesn't strike me as terribly
> > specialized, if one thinks of it more generally as a
> > "derivation/relationship" Attribute.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > For additional commands, e-mail: dev-h...@lucene.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
> --
CTO, OpenSource Connections
Author, Relevant Search
http://o19s.com/doug


Re: SynonymQuery / Query Expansion Strategies Discussion

2018-11-28 Thread Alan Woodward
I think we can expose this information now with a small tweak to the 
SynonymGraphFilter, using the already-existing TypeAttribute.

SGF is hard-coded to set the type attribute to “SYNONYM” on all tokens that it 
inserts into the stream.  It should be simple to add another constructor 
parameter allowing users to change this; then you can chain synonym filters, 
one for each type of expansion you want: synonym, hyponym, hypernym, whatever, 
each setting the type attribute differently.

> On 28 Nov 2018, at 15:59, Michael Gibney  wrote:
> 
> I think the objection to "boosting" in token filters isn't because it
> is "too much", but rather because it breaks the abstraction of the
> analysis chain to directly target scoring (as implied by
> characterizing as "boosting").
> 
> That said, I'm sympathetic to an approach that would establish an
> Attribute to expose the kind of information that would be useful in
> the context of synonyms (or other sorts of derived tokens discussed
> here, where it could be useful to express information about token
> derivation). Such an Attribute would not be directly related to
> scoring/boosting, but would be related to analysis per se, (e.g.,
> source token text, thesaurus, degree of confidence, etc.); support
> could be selectively implemented by TokenFilters, and optionally
> leveraged by query builders (e.g., translated to boosts) or even
> recorded to index Payloads by a final custom analysis component 
> 
> "You can look at any attribute on the tokenstream you want", "rely on
> abstract attributes (type, ...) then it should be easy to sub-class
> the query builder to access them".  Obviously that works iff analysis
> components record the relevant information in attributes on the
> tokenstream, which I think they currently don't (for much of the
> information that has been discussed here) ... and I know of no
> standard way to express the relevant information on the tokenstream.
> 
> I can see that such an Attribute would be out of place (too
> specialized) in the context of the Attributes in lucene/core; but
> there are lots of more specialized Attributes in the various
> submodules under lucene/analysis/* (SynonymGraphFilter lives in
> analysis-common, FWIW). Again, this doesn't strike me as terribly
> specialized, if one thinks of it more generally as a
> "derivation/relationship" Attribute.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: SOLR: Unnecessary logging

2018-11-28 Thread Gus Heck
> There's already LOG4J_PROPS so if you have a favorite log4j config you
> want to use, just set that env variable and forget about it, does that
> work?
>
>
That sounds great, now if only it were in the docs :)
https://lucene.apache.org/solr/guide/7_5/configuring-logging.html

I'll add it later if nobody beats me to it.

-Gus

-- 
http://www.the111shift.com


[jira] [Resolved] (LUCENE-8529) Use the completion key to tiebreak completion suggestion

2018-11-28 Thread Jim Ferenczi (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jim Ferenczi resolved LUCENE-8529.
--
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

> Use the completion key to tiebreak completion suggestion
> 
>
> Key: LUCENE-8529
> URL: https://issues.apache.org/jira/browse/LUCENE-8529
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Fix For: master (8.0), 7.7
>
> Attachments: LUCENE-8529.patch
>
>
> Today the completion suggester uses the document id to tiebreak completion 
> suggestion with same scores. It would improve the stability of the sort to 
> use the surface form of suggestions as the first tiebreaker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk-9) - Build # 959 - Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/959/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly

Error Message:
Unexpected number of elements in the group for intGSF: 8

Stack Trace:
java.lang.AssertionError: Unexpected number of elements in the group for 
intGSF: 8
at 
__randomizedtesting.SeedInfo.seed([4A5A9F488F54BE51:D1E1F110C20C8C0F]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.DocValuesNotIndexedTest.testGroupingDVOnly(DocValuesNotIndexedTest.java:381)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 12517 lines...]
   [junit4] Suite: org.apache.solr.cloud.DocValuesNotIndexedTest
   [junit4]   2> 27181 INFO  
(SUITE-DocValuesNotIndexedTest-seed#[4A5A9F488F54BE51]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & 

[jira] [Commented] (LUCENE-8529) Use the completion key to tiebreak completion suggestion

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702145#comment-16702145
 ] 

ASF subversion and git services commented on LUCENE-8529:
-

Commit 5e2db9eca3bcb897673855ef9392735ab6c64186 in lucene-solr's branch 
refs/heads/branch_7x from [~jim.ferenczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5e2db9e ]

LUCENE-8529: TopSuggestDocsCollector will now use the completion key to 
tiebreak completion
suggestions with identical scores.


> Use the completion key to tiebreak completion suggestion
> 
>
> Key: LUCENE-8529
> URL: https://issues.apache.org/jira/browse/LUCENE-8529
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8529.patch
>
>
> Today the completion suggester uses the document id to tiebreak completion 
> suggestion with same scores. It would improve the stability of the sort to 
> use the surface form of suggestions as the first tiebreaker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702135#comment-16702135
 ] 

Rinka Singh edited comment on LUCENE-7745 at 11/28/18 4:57 PM:
---

Edited.  Sorry...

A few questions.
* How critical is the inverted index to the user experience?
* What happens if the inverted index is speeded up?
* How many AWS instances would usually be used for searching through ~140GB 
sized inverted index and are there any performance numbers around this? (I'd 
like to compare to a server with 8 GPUs costing about $135-140K) - not sure 
what the equivalent GPU instances on Google Cloud/AWS would cost... 

Assumptions (please validate):
 * Documents are being added to the inverted index however the Index itself 
doesn't grow rapidly
 * the Maximum Index size will be less than 140GB - I assume 8 GPUs


was (Author: rinka):
A few questions.  How critical is the inverted index to the user experience?  
What happens if the inverted index is speeded up?

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702135#comment-16702135
 ] 

Rinka Singh commented on LUCENE-7745:
-

A few questions.  How critical is the inverted index to the user experience?  
What happens if the inverted index is speeded up?

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.4) - Build # 3154 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3154/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseG1GC

8 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at 
https://127.0.0.1:43095/solr/collection1_shard2_replica_n2: Expected mime type 
application/octet-stream but got text/html.Error 404 
Can not find: /solr/collection1_shard2_replica_n2/update  
HTTP ERROR 404 Problem accessing 
/solr/collection1_shard2_replica_n2/update. Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at https://127.0.0.1:43095/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([899505F5BBA8EEC:CA2E6C3758FA7E94]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-SmokeRelease-7.6 - Build # 15 - Still Failing

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.6/15/

No tests ran.

Build Log:
[...truncated 23438 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2443 links (1994 relative) to 3203 anchors in 247 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/solr/package/solr-7.6.0.tgz
 into 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.6/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:

[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Toke Eskildsen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702126#comment-16702126
 ] 

Toke Eskildsen commented on SOLR-13013:
---

Ah! I understand now, thanks. Guess I got a bit too focused on index data IO.

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold 
> the whole sliding window scaled result set in memory. This might well be a 
> showstopper as there is no real limit to how large this partial result set 
> can be. Maybe such an optimization could be requested explicitly if the user 
> knows that there is enough memory?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8529) Use the completion key to tiebreak completion suggestion

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702111#comment-16702111
 ] 

ASF subversion and git services commented on LUCENE-8529:
-

Commit 81c092d8262a68dfda3994e790f2e1f3fdf275e2 in lucene-solr's branch 
refs/heads/master from [~jim.ferenczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=81c092d ]

LUCENE-8529: TopSuggestDocsCollector will now use the completion key to 
tiebreak completion
suggestions with identical scores.


> Use the completion key to tiebreak completion suggestion
> 
>
> Key: LUCENE-8529
> URL: https://issues.apache.org/jira/browse/LUCENE-8529
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8529.patch
>
>
> Today the completion suggester uses the document id to tiebreak completion 
> suggestion with same scores. It would improve the stability of the sort to 
> use the surface form of suggestions as the first tiebreaker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Đạt Cao Mạnh
Hi Uwe,

To enable Conscrypt we have include several related Jetty libraries, the
same thing will need to be done if we uses Java 9 ALPN. So support both
security providers and testing them is kinda painful in my point of view.


On Wed, Nov 28, 2018 at 4:16 PM Đạt Cao Mạnh 
wrote:

> Hi folks, Uwe
>
> After upgrading to Conscrypt 1.4.1. I still failure on
> https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/7/consoleText. I
> kinda feel that this SSL is big pain but can be solved easily in Java 9.
> Should it is possible for us to not supporting SSL in HTTP2 (anyone want to
> run SSL can use any HTTP2 proxy like nginx as a gate keeper).
>
> Thanks!
>
> On Wed, Nov 28, 2018 at 10:55 AM Uwe Schindler  wrote:
>
>> Hi,
>>
>>
>>
>> About the bundling of conscrypt.jar in the binary distribution of Apache
>> Solr, I opened LEGAL-425:
>>
>> https://issues.apache.org/jira/browse/LEGAL-425
>>
>>
>>
>> Uwe
>>
>>
>>
>> -
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> http://www.thetaphi.de
>>
>> eMail: u...@thetaphi.de
>>
>>
>>
>> *From:* Uwe Schindler 
>> *Sent:* Wednesday, November 28, 2018 11:38 AM
>> *To:* dev@lucene.apache.org
>> *Subject:* RE: Poll: Merge jira/http2 to master branch
>>
>>
>>
>> Hi Dat, hi Shawn,
>>
>>
>>
>> Thanks for the confirmation! This is what I had in mind (missing ALPN
>> support).
>>
>>
>>
>> IMHO, we should not yet switch away from Java 8 as minimum requirement.
>> With multi-release JARs we are at the moment perfectly able to handle users
>> with newer Java versions, but we should still support Java 8 (as its still
>> widely deployed), although EOL is close. As Redhat, AdoptOpenJDK, and
>> several Linux distributions still provide support for Java 8 at least for 2
>> or 3 more years, I think it’s to early to switch. With recent changes, the
>> support by Oracle is no longer the way to go, as there are many
>> alternatives.
>>
>>
>>
>> My proposal would be to wait until master is branched to “branch_8x”
>> (stays on Java 8 + MR-JAR support), then change “master” to have Java 11 as
>> minimum requirement.
>>
>>
>>
>> About HTTP2: I have no problem with bundling conscrypt (is it needed for
>> both Jetty Server and Jetty Client)?, but disable it by default and only
>> register it in the Java TLS SPI, if Java 8 is used. On Java 9 and later use
>> the shipped HTTP2 support. My proposal would be to do it with a
>> Multirelease JAR (this is currently enabled in Lucene already).
>>
>>
>>
>> If there are license problems, we can do the following: Disable HTTP2 on
>> Java 8 completely but provide documentation in the Solr Ref Guide how to
>> enable it (something like: download conscrypt-uber.jar from maven and save
>> in solr/lib directory). We can enable it automatically, if class.forName()
>> does not throw CNFE). Maybe add a sysprop, but that’s not needed. I think
>> enabling conscrypt is easy with reflection only, or do we need to compile
>> hard against it. From what I figured out, it’s only used at a few places.
>>
>>
>>
>> SolrJ should by default ship without conscyrpt (make it “optional” maven
>> dependency). If it’s in classpath, enable it.
>>
>>
>>
>> Uwe
>>
>>
>>
>> -
>>
>> Uwe Schindler
>>
>> Achterdiek 19, D-28357 Bremen
>>
>> http://www.thetaphi.de
>>
>> eMail: u...@thetaphi.de
>>
>>
>>
>> *From:* Đạt Cao Mạnh 
>> *Sent:* Wednesday, November 28, 2018 11:01 AM
>> *To:* Solr/Lucene Dev 
>> *Subject:* Re: Poll: Merge jira/http2 to master branch
>>
>>
>>
>> Hi,
>>
>>
>>
>> I will try to summary all the options here
>>
>> 1. No support for HTTP/2 + SSL at all, if users want to run SSL they must
>> stick with HTTP/1.1
>>
>> 2. Set Java requirement to 9 and use ALPN implementation of JDK 9
>>
>> 3. If users want to use HTTP/2, we will notice about license problem then
>> download and use Conscrypt library. Of course that we still test the
>> Conscrypt implementation in tests.
>>
>>
>>
>>
>>
>> On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey  wrote:
>>
>> On 11/27/2018 3:32 AM, Uwe Schindler wrote:
>> > I'd like to bring one thing into attention: This library conscrypt is
>> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork
>> named BoringSSL. I'd recommend to check the licensing, because OpenSSL
>> licenses are a bit strange (some BSD fork, also ASF2, but some code is
>> still outdated - I am not sure what the fork actually uses). I have the
>> feeling we should include ASF legal department.
>> >
>> > Nevertheless, I am not 100% sure if conscrypt should really be inclued
>> by default in SolrJ. As SolrJ is a client library for Solr and can be used
>> by external projects, there is the problem of incompatibilities with the
>> native C code included. E.g., if one uses SolrJ with IBM J9 or other Java
>> versions different from openjdk. So I'd be careful. My question is: Do we
>> really need that library. To me it looks like it improves speed only, or is
>> there something that requires its use?
>>
>> As you might know ... full and proper 

[jira] [Created] (SOLR-13021) You cannot safely create and delete the same collection.

2018-11-28 Thread Mark Miller (JIRA)
Mark Miller created SOLR-13021:
--

 Summary: You cannot safely create and delete the same collection.
 Key: SOLR-13021
 URL: https://issues.apache.org/jira/browse/SOLR-13021
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Mark Miller






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Đạt Cao Mạnh
Hi folks, Uwe

After upgrading to Conscrypt 1.4.1. I still failure on
https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/7/consoleText. I
kinda feel that this SSL is big pain but can be solved easily in Java 9.
Should it is possible for us to not supporting SSL in HTTP2 (anyone want to
run SSL can use any HTTP2 proxy like nginx as a gate keeper).

Thanks!

On Wed, Nov 28, 2018 at 10:55 AM Uwe Schindler  wrote:

> Hi,
>
>
>
> About the bundling of conscrypt.jar in the binary distribution of Apache
> Solr, I opened LEGAL-425:
>
> https://issues.apache.org/jira/browse/LEGAL-425
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> http://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Uwe Schindler 
> *Sent:* Wednesday, November 28, 2018 11:38 AM
> *To:* dev@lucene.apache.org
> *Subject:* RE: Poll: Merge jira/http2 to master branch
>
>
>
> Hi Dat, hi Shawn,
>
>
>
> Thanks for the confirmation! This is what I had in mind (missing ALPN
> support).
>
>
>
> IMHO, we should not yet switch away from Java 8 as minimum requirement.
> With multi-release JARs we are at the moment perfectly able to handle users
> with newer Java versions, but we should still support Java 8 (as its still
> widely deployed), although EOL is close. As Redhat, AdoptOpenJDK, and
> several Linux distributions still provide support for Java 8 at least for 2
> or 3 more years, I think it’s to early to switch. With recent changes, the
> support by Oracle is no longer the way to go, as there are many
> alternatives.
>
>
>
> My proposal would be to wait until master is branched to “branch_8x”
> (stays on Java 8 + MR-JAR support), then change “master” to have Java 11 as
> minimum requirement.
>
>
>
> About HTTP2: I have no problem with bundling conscrypt (is it needed for
> both Jetty Server and Jetty Client)?, but disable it by default and only
> register it in the Java TLS SPI, if Java 8 is used. On Java 9 and later use
> the shipped HTTP2 support. My proposal would be to do it with a
> Multirelease JAR (this is currently enabled in Lucene already).
>
>
>
> If there are license problems, we can do the following: Disable HTTP2 on
> Java 8 completely but provide documentation in the Solr Ref Guide how to
> enable it (something like: download conscrypt-uber.jar from maven and save
> in solr/lib directory). We can enable it automatically, if class.forName()
> does not throw CNFE). Maybe add a sysprop, but that’s not needed. I think
> enabling conscrypt is easy with reflection only, or do we need to compile
> hard against it. From what I figured out, it’s only used at a few places.
>
>
>
> SolrJ should by default ship without conscyrpt (make it “optional” maven
> dependency). If it’s in classpath, enable it.
>
>
>
> Uwe
>
>
>
> -
>
> Uwe Schindler
>
> Achterdiek 19, D-28357 Bremen
>
> http://www.thetaphi.de
>
> eMail: u...@thetaphi.de
>
>
>
> *From:* Đạt Cao Mạnh 
> *Sent:* Wednesday, November 28, 2018 11:01 AM
> *To:* Solr/Lucene Dev 
> *Subject:* Re: Poll: Merge jira/http2 to master branch
>
>
>
> Hi,
>
>
>
> I will try to summary all the options here
>
> 1. No support for HTTP/2 + SSL at all, if users want to run SSL they must
> stick with HTTP/1.1
>
> 2. Set Java requirement to 9 and use ALPN implementation of JDK 9
>
> 3. If users want to use HTTP/2, we will notice about license problem then
> download and use Conscrypt library. Of course that we still test the
> Conscrypt implementation in tests.
>
>
>
>
>
> On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey  wrote:
>
> On 11/27/2018 3:32 AM, Uwe Schindler wrote:
> > I'd like to bring one thing into attention: This library conscrypt is
> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork
> named BoringSSL. I'd recommend to check the licensing, because OpenSSL
> licenses are a bit strange (some BSD fork, also ASF2, but some code is
> still outdated - I am not sure what the fork actually uses). I have the
> feeling we should include ASF legal department.
> >
> > Nevertheless, I am not 100% sure if conscrypt should really be inclued
> by default in SolrJ. As SolrJ is a client library for Solr and can be used
> by external projects, there is the problem of incompatibilities with the
> native C code included. E.g., if one uses SolrJ with IBM J9 or other Java
> versions different from openjdk. So I'd be careful. My question is: Do we
> really need that library. To me it looks like it improves speed only, or is
> there something that requires its use?
>
> As you might know ... full and proper http/2 support with Java 8
> requires the conscrypt library.  With Java 9 or higher, the
> functionality is provided natively by Java.  If I remember right, what
> conscrypt provides that Java 8 lacks is ALPN.
>
> If there are license issues with conscrypt, perhaps it might make sense
> to require a newer major version of Java instead ... but I think that
> upgrading the project's Java requirement to 9, 10, or 11 probably
> requires a wider 

Re: SynonymQuery / Query Expansion Strategies Discussion

2018-11-28 Thread Michael Gibney
I think the objection to "boosting" in token filters isn't because it
is "too much", but rather because it breaks the abstraction of the
analysis chain to directly target scoring (as implied by
characterizing as "boosting").

That said, I'm sympathetic to an approach that would establish an
Attribute to expose the kind of information that would be useful in
the context of synonyms (or other sorts of derived tokens discussed
here, where it could be useful to express information about token
derivation). Such an Attribute would not be directly related to
scoring/boosting, but would be related to analysis per se, (e.g.,
source token text, thesaurus, degree of confidence, etc.); support
could be selectively implemented by TokenFilters, and optionally
leveraged by query builders (e.g., translated to boosts) or even
recorded to index Payloads by a final custom analysis component 

"You can look at any attribute on the tokenstream you want", "rely on
abstract attributes (type, ...) then it should be easy to sub-class
the query builder to access them".  Obviously that works iff analysis
components record the relevant information in attributes on the
tokenstream, which I think they currently don't (for much of the
information that has been discussed here) ... and I know of no
standard way to express the relevant information on the tokenstream.

I can see that such an Attribute would be out of place (too
specialized) in the context of the Attributes in lucene/core; but
there are lots of more specialized Attributes in the various
submodules under lucene/analysis/* (SynonymGraphFilter lives in
analysis-common, FWIW). Again, this doesn't strike me as terribly
specialized, if one thinks of it more generally as a
"derivation/relationship" Attribute.

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: SOLR: Unnecessary logging

2018-11-28 Thread Shawn Heisey

On 11/27/2018 7:29 AM, Gus Heck wrote:
Startup logging that happens just on startup should err on the side of 
verbosity.


I think this is a good policy.

As I said before, I really do applaud the efforts to shrink the logs and 
eliminate cruft.  But I don't think that the motivation and goal should 
just be "short logs".  It should be about ensuring that the only thing 
being logged by default is information that most users will find to be 
relevant, and moving other logs to less severe logging levels.


The default logging should provide information that helps out in 
something I like to think of as "typical troubleshooting". Startup and 
reload logging can be a fairly verbose.  Unless the user is doing 
something very odd, that information is not likely to be logged frequently.


In some cases, achieving the goal I've outlined might mean *ADDING* some 
logging to the default level, to provide information about unexpected 
situations.


The logging that typically happens during normal operation (queries and 
updates, mostly) only really gets verbose if the server is busy.  I 
don't know that there's much cruft in that logging, but we can review it 
just to make sure.  Giving the user the option to push request logging 
into a separate logfile would do a lot to reduce the size of the main 
log.  Troubleshooting is sometimes more difficult when logs are 
separated, but if that becomes an issue the user can always reconfigure 
to put it all back in solr.log.


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702009#comment-16702009
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit aef5a75e1377af41da26dd314b1c0496672d268c in lucene-solr's branch 
refs/heads/branch_7_6 from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=aef5a75 ]

SOLR-12398: retroactively add CHANGES.txt back-compat break for 7.5

(cherry picked from commit 2611c22b6b00d73f6ae1fc6996e029f73ee7458d)


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Fix For: 7.5
>
> Attachments: SOLR-12398.patch, SOLR-12398.patch, 
> repro-facet-heatmaps-response-format-change.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Joel Bernstein (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702034#comment-16702034
 ] 

Joel Bernstein edited comment on SOLR-13013 at 11/28/18 3:39 PM:
-

You're exactly right that improving performance of export helps the MapReduce 
use cases as well. It's just that in a sharded, replicated environment with a 
tier of worker nodes performing a reduce operation, you can get massive 
throughput already just because you can have dozens of servers pushing out an 
export and reducing in parallel.

But you could easily argue that your usecase is the more common use case and we 
should really try to make it as fast as possible.

I wouldn't worry too much about testing this is in sharded scenarios. We can 
extrapolate the single shard findings to multiple shards, realizing that the 
aggregator node will quickly become the bottleneck and the /export will spend 
much of it's time blocked while writing data. Having a tier of worker nodes 
unlocks this bottleneck in the case where worker nodes are performing some form 
of reduce operation.

 


was (Author: joel.bernstein):
 

You're exactly right that improving performance of export helps the MapReduce 
use cases as well. It's just that in a sharded, replicated environment with a 
tier of worker nodes performing a reduce operation, you can get massive 
throughput already just because you can have dozens of servers pushing out an 
export and reducing in parallel.

But you could easily argue that your usecase is the more common use case and we 
should really try to make it as fast as possible.

I wouldn't worry too much about testing this is in sharded scenarios. We can 
extrapolate the single shard findings to multiple shards, realizing that the 
aggregator node will quickly become the bottleneck and the /export will spend 
much of it's time blocked while writing data. Having a tier of worker nodes 
unlocks this bottleneck in the case where worker nodes are performing some form 
of reduce operation.

 

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold 
> the whole sliding window scaled result set in memory. This might well be a 
> showstopper as there is no real limit to how large this partial result set 
> can be. Maybe such an optimization could be requested explicitly if the user 
> knows that there is enough memory?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Joel Bernstein (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702034#comment-16702034
 ] 

Joel Bernstein commented on SOLR-13013:
---

 

You're exactly right that improving performance of export helps the MapReduce 
use cases as well. It's just that in a sharded, replicated environment with a 
tier of worker nodes performing a reduce operation, you can get massive 
throughput already just because you can have dozens of servers pushing out an 
export and reducing in parallel.

But you could easily argue that your usecase is the more common use case and we 
should really try to make it as fast as possible.

I wouldn't worry too much about testing this is in sharded scenarios. We can 
extrapolate the single shard findings to multiple shards, realizing that the 
aggregator node will quickly become the bottleneck and the /export will spend 
much of it's time blocked while writing data. Having a tier of worker nodes 
unlocks this bottleneck in the case where worker nodes are performing some form 
of reduce operation.

 

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold 
> the whole sliding window scaled result set in memory. This might well be a 
> showstopper as there is no real limit to how large this partial result set 
> can be. Maybe such an optimization could be requested explicitly if the user 
> knows that there is enough memory?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



Re: Solr Test Fixing Update

2018-11-28 Thread David Smiley
Thanks for your huge efforts on this Mark!
On Tue, Nov 27, 2018 at 12:26 PM Mark Miller  wrote:

> Just spent a lot of time traveling, but I'm just about ready to make my
> first large commit towards fixing the Solr tests.
>
> First I'll commit to master and then in a day or two I'll commit to 7x as
> well.
>
> This commit will be a huge step forward, but it is still just the start.
>
> ant test should work a lot better for everyone (even without excluding
> badapples), but not every test has been beasted and made solid. I expect
> Jenkins email failures to reduce but not go away yet at all. This commit
> likely fixes hundreds of possible fails, but many long tail fails remain.
>
> I will start tackling the full hardening effort by package (hopefully with
> other volunteers) as the next step.
>
> There is a lot to do beyond that to finish and maintain solid tests, but I
> have a plan that I will continue to execute on that I think we get the
> project there.
>
> - Mark
>
-- 
Lucene/Solr Search Committer (PMC), Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com


[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702006#comment-16702006
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit 2611c22b6b00d73f6ae1fc6996e029f73ee7458d in lucene-solr's branch 
refs/heads/master from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2611c22 ]

SOLR-12398: retroactively add CHANGES.txt back-compat break for 7.5


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Fix For: 7.5
>
> Attachments: SOLR-12398.patch, SOLR-12398.patch, 
> repro-facet-heatmaps-response-format-change.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (LUCENE-8575) Improve toString() in SegmentInfo

2018-11-28 Thread Namgyu Kim (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701982#comment-16701982
 ] 

Namgyu Kim edited comment on LUCENE-8575 at 11/28/18 3:24 PM:
--

Thank you for your reply, [~jpountz] :D

I uploaded a new patch that reflected your opinion.

 

before:
 
TEST(8.0.0):C1:[indexSort=]:[diagnostics=,]:[attributes=,]

after :
 TEST(8.0.0):C1:[indexSort=]:[diagnostics=\{key1=value1, 
key2=value2}]:[attributes=\{key1=value1, key2=value2}]


was (Author: danmuzi):
Thank you for your reply, [~jpountz] :D

I uploaded a new patch that reflected your opinion.

 

before:
TEST(8.0.0):C1:[indexSort=]:[diagnostics=,]:[attributes=,]

after :
TEST(8.0.0):C1:[indexSort=]:[diagnostics=\{key1=value1, 
key2=value2}]:[attributes=\{key1=value1, key2=value2}]

> Improve toString() in SegmentInfo
> -
>
> Key: LUCENE-8575
> URL: https://issues.apache.org/jira/browse/LUCENE-8575
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Reporter: Namgyu Kim
>Priority: Major
> Attachments: LUCENE-8575.patch, LUCENE-8575.patch
>
>
> I saw the following code in SegmentInfo class.
> {code:java}
> // TODO: we could append toString of attributes() here?
> {code}
> Of course, we can.
>  
> So I wrote a code for that part.
> {code:java}
> public String toString(int delCount) {
>   StringBuilder s = new StringBuilder();
>   s.append(name).append('(').append(version == null ? "?" : 
> version).append(')').append(':');
>   char cfs = getUseCompoundFile() ? 'c' : 'C';
>   s.append(cfs);
>   s.append(maxDoc);
>   if (delCount != 0) {
> s.append('/').append(delCount);
>   }
>   if (indexSort != null) {
> s.append(":[indexSort=");
> s.append(indexSort);
> s.append(']');
>   }
>   // New Code
>   if (!diagnostics.isEmpty()) {
> s.append(":[diagnostics=");
> for (Map.Entry entry : diagnostics.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   // New Code
>   if (!attributes.isEmpty()) {
> s.append(":[attributes=");
> for (Map.Entry entry : attributes.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   return s.toString();
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-12398) Make JSON Facet API support Heatmap Facet

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-12398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16702007#comment-16702007
 ] 

ASF subversion and git services commented on SOLR-12398:


Commit e466b007619b9f1235ab87a6c029e6da3a5fb9e8 in lucene-solr's branch 
refs/heads/branch_7x from [~dsmiley]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=e466b00 ]

SOLR-12398: retroactively add CHANGES.txt back-compat break for 7.5

(cherry picked from commit 2611c22b6b00d73f6ae1fc6996e029f73ee7458d)


> Make JSON Facet API support Heatmap Facet
> -
>
> Key: SOLR-12398
> URL: https://issues.apache.org/jira/browse/SOLR-12398
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module, faceting, JSON Request API, spatial
>Reporter: Jaime Yap
>Assignee: David Smiley
>Priority: Major
>  Labels: heatmap
> Fix For: 7.5
>
> Attachments: SOLR-12398.patch, SOLR-12398.patch, 
> repro-facet-heatmaps-response-format-change.patch
>
>
> The JSON query Facet API does not support Heatmap facets. For companies that 
> have standardized around generating queries for the JSON query API, it is a 
> major wart to need to also support falling back to the param encoding API in 
> order to make use of them.
> More importantly however, given it's more natural support for nested 
> subfacets, the JSON Query facet API is be able to compute more interesting 
> Heatmap layers for each facet bucket. Without resorting to the older (and 
> much more awkward) facet pivot syntax.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8575) Improve toString() in SegmentInfo

2018-11-28 Thread Namgyu Kim (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701982#comment-16701982
 ] 

Namgyu Kim commented on LUCENE-8575:


Thank you for your reply, [~jpountz] :D

I uploaded a new patch that reflected your opinion.

 

before:
TEST(8.0.0):C1:[indexSort=]:[diagnostics=,]:[attributes=,]

after :
TEST(8.0.0):C1:[indexSort=]:[diagnostics=\{key1=value1, 
key2=value2}]:[attributes=\{key1=value1, key2=value2}]

> Improve toString() in SegmentInfo
> -
>
> Key: LUCENE-8575
> URL: https://issues.apache.org/jira/browse/LUCENE-8575
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Reporter: Namgyu Kim
>Priority: Major
> Attachments: LUCENE-8575.patch, LUCENE-8575.patch
>
>
> I saw the following code in SegmentInfo class.
> {code:java}
> // TODO: we could append toString of attributes() here?
> {code}
> Of course, we can.
>  
> So I wrote a code for that part.
> {code:java}
> public String toString(int delCount) {
>   StringBuilder s = new StringBuilder();
>   s.append(name).append('(').append(version == null ? "?" : 
> version).append(')').append(':');
>   char cfs = getUseCompoundFile() ? 'c' : 'C';
>   s.append(cfs);
>   s.append(maxDoc);
>   if (delCount != 0) {
> s.append('/').append(delCount);
>   }
>   if (indexSort != null) {
> s.append(":[indexSort=");
> s.append(indexSort);
> s.append(']');
>   }
>   // New Code
>   if (!diagnostics.isEmpty()) {
> s.append(":[diagnostics=");
> for (Map.Entry entry : diagnostics.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   // New Code
>   if (!attributes.isEmpty()) {
> s.append(":[attributes=");
> for (Map.Entry entry : attributes.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   return s.toString();
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701990#comment-16701990
 ] 

Rinka Singh edited comment on LUCENE-7745 at 11/28/18 3:08 PM:
---

[~jpountz]
{quote}(Unrelated to your comment Rinka, but seeing activity on this issue 
reminded me that I wanted to share something) There are limited use-cases for 
GPU accelelation in Lucene due to the fact that query processing is full of 
branches, especially since we added support for impacts and WAND.{quote}

While Yes branches do impact the performance, well designed (GPU) code will 
consist of a combo of both CPU (the decision making part) and GPU code.  For 
example, I wrote a histogram as a test case that saw SIGNIFICANT acceleration 
and I also identified further code areas that can be improved.  I'm fairly sure 
(gut feel), I can squeeze out a 40-50x kind of improvement at the very least on 
a mid-sized GPU (given the time etc.,). I think things will be much, much 
better on a high end GPU and with further scale-up on a multi-gpu system...

My point is - thinking (GPU) parallel is a completely different ball-game and 
requires a mind-shift.  Once that happens, the value add will be massive and 
gut tells me Lucene is a huge opportunity.

Incidentally, this is why I want to develop a library that I can put out there 
for integration.

{quote}That said Mike initially mentioned that BooleanScorer might be one 
scorer that could benefit from GPU acceleration as it scores large blocks of 
documents at once. I just attached a specialization of a disjunction over term 
queries that should make it easy to experiment with Cuda, see the TODO in the 
end on top of the computeScores method.
{quote}

Lucene is really new to me (and so is working with Apache - sorry, I am a 
newbie to Apache) :). Please will you put links here...  


was (Author: rinka):
[~jpountz]
{quote}(Unrelated to your comment Rinka, but seeing activity on this issue 
reminded me that I wanted to share something) There are limited use-cases for 
GPU accelelation in Lucene due to the fact that query processing is full of 
branches, especially since we added support for impacts and WAND.{quote}

While Yes branches do impact the performance, well designed (GPU) code will 
consist of a combo of both CPU (the decision making part) and GPU code.  For 
example, I wrote a histogram as a test case that saw SIGNIFICANT acceleration 
and I also identified further code areas that can be improved.  I'm fairly sure 
(gut feel), I can squeeze out a 40-50x kind of improvement at the very least on 
a mid-sized GPU (given the time etc.,). I think things will be much, much 
better on a high end GPU and with further scale-up on a multi-gpu system...

Incidentally, this is why I want to develop a library that I can put out there 
for integration.

{quote}That said Mike initially mentioned that BooleanScorer might be one 
scorer that could benefit from GPU acceleration as it scores large blocks of 
documents at once. I just attached a specialization of a disjunction over term 
queries that should make it easy to experiment with Cuda, see the TODO in the 
end on top of the computeScores method.
{quote}

Lucene is really new to me (and so is working with Apache - sorry, I am a 
newbie to Apache) :). Please will you put links here...  

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-13008) JSON Document Transformer doesn't heed "indent" parameter

2018-11-28 Thread Eric Pugh (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701991#comment-16701991
 ] 

Eric Pugh commented on SOLR-13008:
--

Thanks [~elyograg] for looking at this.   I was thinking as I worked on the 
patch, of also expanding the documentation.  It's a really powerful feature 
that isn't well known!  Do you think that should be a separate SOLR issue, or 
add additional documentation, including the `indent=true` impact as part of 
this issue?   

As far as the precommit, I did get the error you suspected.   So I added the 
SHA checksum file.   However, reading through the 
`./solr/licenses/README.committers.txt` file, it appears I should add a 
`jackson-dataformat-xml-LICENSE-ASL.txt` and an empty 
`jackson-dataformat-xml-NOTICE.txt` file?   Or is adding that the 
responsibility of the commiter who actually commits this patch?

> JSON Document Transformer doesn't heed "indent" parameter
> -
>
> Key: SOLR-13008
> URL: https://issues.apache.org/jira/browse/SOLR-13008
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Response Writers
>Affects Versions: 7.5
>Reporter: Eric Pugh
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> A query like "wt=json=id,subject:[json]=true", will return the 
> field subject as JSON, however the indent parameter is ignored on the nested 
> JSON.   This can lead to a very wide response that you have to scroll over.  
> Often I add indent=true becasue I want to see the structure of my embedded 
> JSON text, and I need to cut'n'paste over to another editor.   This would let 
> the nested JSON object be pretty printed.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701990#comment-16701990
 ] 

Rinka Singh commented on LUCENE-7745:
-

[~jpountz]
{quote}(Unrelated to your comment Rinka, but seeing activity on this issue 
reminded me that I wanted to share something) There are limited use-cases for 
GPU accelelation in Lucene due to the fact that query processing is full of 
branches, especially since we added support for impacts and WAND.{quote}

While Yes branches do impact the performance, well designed (GPU) code will 
consist of a combo of both CPU (the decision making part) and GPU code.  For 
example, I wrote a histogram as a test case that saw SIGNIFICANT acceleration 
and I also identified further code areas that can be improved.  I'm fairly sure 
(gut feel), I can squeeze out a 40-50x kind of improvement at the very least on 
a mid-sized GPU (given the time etc.,). I think things will be much, much 
better on a high end GPU and with further scale-up on a multi-gpu system...

Incidentally, this is why I want to develop a library that I can put out there 
for integration.

{quote}That said Mike initially mentioned that BooleanScorer might be one 
scorer that could benefit from GPU acceleration as it scores large blocks of 
documents at once. I just attached a specialization of a disjunction over term 
queries that should make it easy to experiment with Cuda, see the TODO in the 
end on top of the computeScores method.
{quote}

Lucene is really new to me (and so is working with Apache - sorry, I am a 
newbie to Apache) :). Please will you put links here...  

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[GitHub] lucene-solr issue #455: SOLR-12638

2018-11-28 Thread dsmiley
Github user dsmiley commented on the issue:

https://github.com/apache/lucene-solr/pull/455
  
What does "so _root_ is not hard-coded into childless documents" mean?  If 
you think SOLR-5211 is missing something, can you please post a patch file to 
the issue and explain what you're adding?


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.6-Linux (64bit/jdk-9.0.4) - Build # 38 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/38/
Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

4 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at 
https://127.0.0.1:35947/solr/collection1_shard2_replica_n2: Expected mime type 
application/octet-stream but got text/html.Error 404 
Can not find: /solr/collection1_shard2_replica_n2/update  
HTTP ERROR 404 Problem accessing 
/solr/collection1_shard2_replica_n2/update. Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at https://127.0.0.1:35947/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([A693A71392D6C188:64249B7B919631F0]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Updated] (LUCENE-8575) Improve toString() in SegmentInfo

2018-11-28 Thread Namgyu Kim (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Namgyu Kim updated LUCENE-8575:
---
Attachment: LUCENE-8575.patch

> Improve toString() in SegmentInfo
> -
>
> Key: LUCENE-8575
> URL: https://issues.apache.org/jira/browse/LUCENE-8575
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Reporter: Namgyu Kim
>Priority: Major
> Attachments: LUCENE-8575.patch, LUCENE-8575.patch
>
>
> I saw the following code in SegmentInfo class.
> {code:java}
> // TODO: we could append toString of attributes() here?
> {code}
> Of course, we can.
>  
> So I wrote a code for that part.
> {code:java}
> public String toString(int delCount) {
>   StringBuilder s = new StringBuilder();
>   s.append(name).append('(').append(version == null ? "?" : 
> version).append(')').append(':');
>   char cfs = getUseCompoundFile() ? 'c' : 'C';
>   s.append(cfs);
>   s.append(maxDoc);
>   if (delCount != 0) {
> s.append('/').append(delCount);
>   }
>   if (indexSort != null) {
> s.append(":[indexSort=");
> s.append(indexSort);
> s.append(']');
>   }
>   // New Code
>   if (!diagnostics.isEmpty()) {
> s.append(":[diagnostics=");
> for (Map.Entry entry : diagnostics.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   // New Code
>   if (!attributes.isEmpty()) {
> s.append(":[attributes=");
> for (Map.Entry entry : attributes.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   return s.toString();
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701976#comment-16701976
 ] 

Rinka Singh commented on LUCENE-7745:
-

> The code is not worth a patch right now, but will soon have something. I 
> shall update on the latest state
> here as soon as I find myself some time (winding down from a hectic Black 
> Friday/Cyber Monday support schedule).

Do you think I could take a look at the code, I could do a quick review and 
perhaps add a bit of value.  I'm fine if the code is in dev state.

Would you have written up something to describe what you are doing?

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[GitHub] lucene-solr pull request #511: Remove k1+1 from the numerator of BM25Similar...

2018-11-28 Thread javanna
GitHub user javanna opened a pull request:

https://github.com/apache/lucene-solr/pull/511

Remove k1+1 from the numerator of  BM25Similarity

Patch for https://issues.apache.org/jira/browse/LUCENE-8563.

This PR removes the k1+1 factor from the numerator of `BM25Similarity` and 
adds a new `LegacyBM25Similarity` under misc that exposes the old behaviour. 
Note that I haven't found a way to easily reproduce the previous behaviour in 
the explain method, so I left that part out of `LegacyBM25Similarity` for now.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/javanna/lucene-solr 
lucene-8563_bm25_k1_numerator

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/511.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #511


commit c665ed04b94f330fe27deec781cc1c59d45fddb5
Author: javanna 
Date:   2018-11-14T09:51:48Z

Remove k1+1 constant factor from BM25 formula numerator

commit 1b3714410771b53d30f29c78f27c37c617dea85c
Author: javanna 
Date:   2018-11-14T11:15:39Z

adapt TestFunctionQuery

commit 7f56226906ededdc285877c429a00dae4f5a6d10
Author: javanna 
Date:   2018-11-14T14:08:38Z

adapt TestPayloadScoreQParserPlugin

commit 870d1d79afef7800ec348fabbe7b14b1a11ae1c0
Author: javanna 
Date:   2018-11-14T16:32:53Z

add migrate note

commit 5f0a0b0fb0ec7ebad8094f7fa97a09c877d3a1cb
Author: javanna 
Date:   2018-11-27T21:15:12Z

add LegacyBM25Similarity




---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

2018-11-28 Thread Christophe Bismuth (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701958#comment-16701958
 ] 

Christophe Bismuth commented on LUCENE-8548:


Thanks a lot for sharing this [~jim.ferenczi], and no worries at all as the 
first iteration was an interesting journey! I think taking time to read about 
Viterbi would help me some more, let's add it to my pretty own todo list :D

I diffed your patch with {{master}} and debugged new tests step-by-step, and I 
think I understand the big picture. Among others, I totally missed the {{if 
(isCommonOrInherited(scriptCode) && isCommonOrInherited(sc) == false)}} 
condition which is essential.

I still have one more question, could you please explain what information is 
contained in the {{wordIdRef}} variable and what the 
{{unkDictionary.lookupWordIds(characterId, wordIdRef)}} statement does? The 
debugger tells me {{wordIdRef.length}} is always equal to 36 or 42 and even 
though 42 is a really great number, I'm a tiny lost in there ...

> Reevaluate scripts boundary break in Nori's tokenizer
> -
>
> Key: LUCENE-8548
> URL: https://issues.apache.org/jira/browse/LUCENE-8548
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8548.patch, screenshot-1.png, 
> testCyrillicWord.dot.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526:
> {noformat}
> Tokens are split on different character POS types (which seem to not quite 
> line up with Unicode character blocks), which leads to weird results for 
> non-CJK tokens:
> εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other 
> symbol) + μί/SL(Foreign language)
> ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + 
> ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol)
> Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + 
> лтичко/SL(Foreign language) + ̄/SY(Other symbol)
> don't is tokenized as don + t; same for don't (with a curly apostrophe).
> אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol)
> Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow
> While it is still possible to find these words using Nori, there are many 
> more chances for false positives when the tokens are split up like this. In 
> particular, individual numbers and combining diacritics are indexed 
> separately (e.g., in the Cyrillic example above), which can lead to a 
> performance hit on large corpora like Wiktionary or Wikipedia.
> Work around: use a character filter to get rid of combining diacritics before 
> Nori processes the text. This doesn't solve the Greek, Hebrew, or English 
> cases, though.
> Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek 
> Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. 
> Combining diacritics should not trigger token splits. Non-CJK text should be 
> tokenized on spaces and punctuation, not by character type shifts. 
> Apostrophe-like characters should not trigger token splits (though I could 
> see someone disagreeing on this one).{noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Ishan Chattopadhyaya (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701951#comment-16701951
 ] 

Ishan Chattopadhyaya commented on LUCENE-7745:
--

bq. Your thoughts please...
Thanks for your interest in this. Seems like your proposed ideas are very much 
inline with our approach that we're trying out as well. There are some initial 
experiments and results that we are performing as we speak, and I can see that 
there are benefits in the niche usecases that Adrien mentioned.

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-6117) Replication command=fetchindex always return success.

2018-11-28 Thread Jason Gerlowski (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701957#comment-16701957
 ] 

Jason Gerlowski commented on SOLR-6117:
---

Attached an updated patch that's intended for the master branch, and thus has 
liberty to do more to make the various responses from the /replication API more 
uniform.  This version of the patch addresses all of the bullet points in my 
previous comment.  Haven't run tests more generally yet, but I hope to commit 
to master in the next week or so.

One thing I forgot to clarify in my previous comment: both of these patches 
address _all_ subcommands in the /replication API (not just "fetchindex")  That 
was a point of discussion in the original effort on this JIRA, so just thought 
I'd clarify.

> Replication command=fetchindex always return success.
> -
>
> Key: SOLR-6117
> URL: https://issues.apache.org/jira/browse/SOLR-6117
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.6, 7.5
>Reporter: Raintung Li
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-6117-master.patch, SOLR-6117.patch, 
> SOLR-6117.patch, SOLR-6117.patch, SOLR-6117.txt
>
>
> Replication API command=fetchindex do fetch the index. while occur the error, 
> still give success response. 
> API should return the right status, especially WAIT parameter is 
> true.(synchronous).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-6117) Replication command=fetchindex always return success.

2018-11-28 Thread Jason Gerlowski (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gerlowski updated SOLR-6117:
--
Attachment: SOLR-6117-master.patch

> Replication command=fetchindex always return success.
> -
>
> Key: SOLR-6117
> URL: https://issues.apache.org/jira/browse/SOLR-6117
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.6, 7.5
>Reporter: Raintung Li
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-6117-master.patch, SOLR-6117.patch, 
> SOLR-6117.patch, SOLR-6117.patch, SOLR-6117.txt
>
>
> Replication API command=fetchindex do fetch the index. while occur the error, 
> still give success response. 
> API should return the right status, especially WAIT parameter is 
> true.(synchronous).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Updated] (SOLR-6117) Replication command=fetchindex always return success.

2018-11-28 Thread Jason Gerlowski (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gerlowski updated SOLR-6117:
--
Affects Version/s: 7.5

> Replication command=fetchindex always return success.
> -
>
> Key: SOLR-6117
> URL: https://issues.apache.org/jira/browse/SOLR-6117
> Project: Solr
>  Issue Type: Bug
>  Components: replication (java)
>Affects Versions: 4.6, 7.5
>Reporter: Raintung Li
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-6117.patch, SOLR-6117.patch, SOLR-6117.patch, 
> SOLR-6117.txt
>
>
> Replication API command=fetchindex do fetch the index. while occur the error, 
> still give success response. 
> API should return the right status, especially WAIT parameter is 
> true.(synchronous).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Toke Eskildsen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701946#comment-16701946
 ] 

Toke Eskildsen commented on SOLR-13013:
---

[~joel.bernstein] Unfortunately I don't have proper hardware at hand to test 
with our large shards in a multi-shard setup. I _could_ put them on a spinning 
drive, now that I think about it, but I am also afraid that my test-box does 
not have adequate memory to fully cache the DocValues structures when using 
multiple shards, so that would complicate testing somewhat. I'll see what else 
we have lying around and if nothing else, I could just delete 3/4th of the data 
in 4 of the shards and run with those instead (takes some days to do though).

Up until now we have used export exclusively to do simple query-bases 
data-dumps, so that was my go-to case. It is probably due to my limited 
understanding of Streaming Expressions that I do not understand the 
methodological problem in my test:

I get that multi-sharding, replicas and hashing (bit unsure about the hashing 
part) can distribute and parallelize the load to make processing faster, but 
only the "more and smaller shards" of those 3 would reduce the total amount of 
work, as I understand it? So with regard to that, any optimization to the 
export should work equally well for a single-shard simple export and a more 
complex distributed setup, measured as total work to be done?

I am on (even) more shaky grounds with the local reduce operation. Isn't that 
step after the export part and therefore extremely dependent on raw export 
speed? Or is there some shortcut mechanism I haven't understood?

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold 
> the whole sliding window scaled result set in memory. This might well be a 
> showstopper as there is no real limit to how large this partial result set 
> can be. Maybe such an optimization could be requested explicitly if the user 
> knows that there is enough memory?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Ishan Chattopadhyaya (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701941#comment-16701941
 ] 

Ishan Chattopadhyaya commented on LUCENE-7745:
--

Hi Rinka, Kishore and I made some progress on this and presented our current 
state of this initiative here: https://www.youtube.com/watch?v=cY_4ApOAVJQ
The code is not worth a patch right now, but will soon have something. I shall 
update on the latest state here as soon as I find myself some time (winding 
down from a hectic Black Friday/Cyber Monday support schedule).


> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Joel Bernstein (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701898#comment-16701898
 ] 

Joel Bernstein edited comment on SOLR-13013 at 11/28/18 1:48 PM:
-

Interesting findings. I can work on getting this patch committed, possibly for 
the 8.0 release.

A couple of thoughts about the design of the /export handler.

The /export handler was very much designed to support MapReduce operations 
(distributed grouping, rollups, relational algebra) in Streaming Expressions. 
Scaling these MapReduce operations took the following path:

1) Sharding: The /export handler benefits tremendously by sharding. The 
benefits go well beyond linear. This is because 2 shards both doubles the 
computing power and more then halves the amount of work that needs to done by 
each shard. 

3) Hash partitioning and worker collections: Sharding very quickly causes 
bottlenecks on a single aggregator node. Streaming Expressions parallel 
function when combined with the hash partitioner allows the /exports to be 
partitioned into X number of slices and brings into play not just the shards 
but the replicas. When a reduce operations happens on the worker nodes 
(rollups, innerJoins) which limits the numbers of records that are emitted in 
the final stream, this is an extremely powerful scaling tool.

So, from a pure /export standpoint with no reduce operation, all from a single 
shard, you are working somewhat against the design goals of the system. But 
that being said the faster we make the pure export form a single shard, the 
more use cases the /export handler serves.


was (Author: joel.bernstein):
Interesting findings. I can work on getting this patch committed, possibly for 
the 8.0 release.

A couple of thoughts about the design of the /export handler.

The /export handler was very much designed to support MapReduce operations 
(distributed grouping, rollups, relational algebra) in Streaming Expressions. 
Scaling these MapReduce operations took the following path:

1) Sharding: The /export handler benefits tremendously by sharding. The 
benefits go well beyond linear. This is because 2 shards both doubles the 
computing power and more then halves the amount of work that needs to done by 
each shard. 

3) Hash partitioning and worker collections: Sharding very quickly causes 
bottlenecks on a single aggregator node. Streaming Expressions parallel 
function when combined with the hash partitioner allows the /exports to be 
partitioned into X number of slices and brings into play not just the shards 
but the replicas. When a reduce operations happens on the worker nodes 
(rollups, innerJoins) which limits the numbers of records that are emitted in 
the final stream, this is an extremely powerful scaling tool.

So, from a pure /export standpoint with no reduce operation, all from a single 
shard, you are working somewhat against the design goals of system. But that 
being said the faster we make the pure export form a single shard, the more use 
cases the /export handler serves.

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to 

[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Joel Bernstein (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701898#comment-16701898
 ] 

Joel Bernstein edited comment on SOLR-13013 at 11/28/18 1:48 PM:
-

Interesting findings. I can work on getting this patch committed, possibly for 
the 8.0 release.

A couple of thoughts about the design of the /export handler.

The /export handler was very much designed to support MapReduce operations 
(distributed grouping, rollups, relational algebra) in Streaming Expressions. 
Scaling these MapReduce operations took the following path:

1) Sharding: The /export handler benefits tremendously by sharding. The 
benefits go well beyond linear. This is because 2 shards both doubles the 
computing power and more then halves the amount of work that needs to done by 
each shard. 

3) Hash partitioning and worker collections: Sharding very quickly causes 
bottlenecks on a single aggregator node. Streaming Expressions parallel 
function when combined with the hash partitioner allows the /exports to be 
partitioned into X number of slices and brings into play not just the shards 
but the replicas. When a reduce operations happens on the worker nodes 
(rollups, innerJoins) which limits the numbers of records that are emitted in 
the final stream, this is an extremely powerful scaling tool.

So, from a pure /export standpoint with no reduce operation, all from a single 
shard, you are working somewhat against the design goals of system. But that 
being said the faster we make the pure export form a single shard, the more use 
cases the /export handler serves.


was (Author: joel.bernstein):
Interesting findings. I can work on getting this patch committed, possibly for 
the 8.0 release.

A couple of thoughts about the design of the /export handler.

The /export handler was very much designed to support MapReduce operations 
(distributed grouping, rollups, relational algebra) in Streaming Expressions. 
Scaling these MapReduce operations took the following path:

1) Sharding: The /export handler benefits tremendously by sharding. The 
benefits go well beyond linear. This is because 2 shards both doubles the 
computing power and more then halves the amount of work that needs to done by 
each shard. 

3) Hash partitioning and worker collections: Sharding very quickly causes 
bottlenecks on a single aggregator node. Streaming Expressions parallel 
function when combined with the hash partitioner allows the /exports to be 
partitioned into X number of slices and brings into play not just the shards 
but the replicas. When a reduce operations happens on the worker nodes 
(rollups, innerJoins) which limits the numbers of records that are emitted in 
the final stream, this is an extremely powerful scaling tool.

So, from a pure /export standpoint with no reduce operation, all from a single 
shard, you are working somewhat against the design goals of system. But that 
being said the faster we make the pure export form a single shard the more use 
cases the the /export handler serves.

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to 

[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Joel Bernstein (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701898#comment-16701898
 ] 

Joel Bernstein commented on SOLR-13013:
---

Interesting findings. I can work on getting this patch committed, possibly for 
the 8.0 release.

A couple of thoughts about the design of the /export handler.

The /export handler was very much designed to support MapReduce operations 
(distributed grouping, rollups, relational algebra) in Streaming Expressions. 
Scaling these MapReduce operations took the following path:

1) Sharding: The /export handler benefits tremendously by sharding. The 
benefits go well beyond linear. This is because 2 shards both doubles the 
computing power and more then halves the amount of work that needs to done by 
each shard. 

3) Hash partitioning and worker collections: Sharding very quickly causes 
bottlenecks on a single aggregator node. Streaming Expressions parallel 
function when combined with the hash partitioner allows the /exports to be 
partitioned into X number of slices and brings into play not just the shards 
but the replicas. When a reduce operations happens on the worker nodes 
(rollups, innerJoins) which limits the numbers of records that are emitted in 
the final stream, this is an extremely powerful scaling tool.

So, from a pure /export standpoint with no reduce operation, all from a single 
shard, you are working somewhat against the design goals of system. But that 
being said the faster we make the pure export form a single shard the more use 
cases the the /export handler serves.

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # Extract the DocValues to a temporary on-heap structure
>  # Re-sort the extracted values to the original sliding window order
> Deliver the values
> One big difference from the current export code is of course the need to hold 
> the whole sliding window scaled result set in memory. This might well be a 
> showstopper as there is no real limit to how large this partial result set 
> can be. Maybe such an optimization could be requested explicitly if the user 
> knows that there is enough memory?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701879#comment-16701879
 ] 

Adrien Grand commented on LUCENE-7745:
--

(Unrelated to your comment Rinka, but seeing activity on this issue reminded me 
that I wanted to share something) There are limited use-cases for GPU 
accelelation in Lucene due to the fact that query processing is full of 
branches, especially since we added support for impacts and WAND. That said 
Mike initially mentioned that BooleanScorer might be one scorer that could 
benefit from GPU acceleration as it scores large blocks of documents at once. I 
just attached a specialization of a disjunction over term queries that should 
make it easy to experiment with Cuda, see the TODO in the end on top of the 
{{computeScores}} method.

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8575) Improve toString() in SegmentInfo

2018-11-28 Thread Adrien Grand (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701886#comment-16701886
 ] 

Adrien Grand commented on LUCENE-8575:
--

Thanks [~danmuzi]. Let's reuse attributes.toString() and diagnostics.toString() 
instead of reimplementing Map#toString?

> Improve toString() in SegmentInfo
> -
>
> Key: LUCENE-8575
> URL: https://issues.apache.org/jira/browse/LUCENE-8575
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Reporter: Namgyu Kim
>Priority: Major
> Attachments: LUCENE-8575.patch
>
>
> I saw the following code in SegmentInfo class.
> {code:java}
> // TODO: we could append toString of attributes() here?
> {code}
> Of course, we can.
>  
> So I wrote a code for that part.
> {code:java}
> public String toString(int delCount) {
>   StringBuilder s = new StringBuilder();
>   s.append(name).append('(').append(version == null ? "?" : 
> version).append(')').append(':');
>   char cfs = getUseCompoundFile() ? 'c' : 'C';
>   s.append(cfs);
>   s.append(maxDoc);
>   if (delCount != 0) {
> s.append('/').append(delCount);
>   }
>   if (indexSort != null) {
> s.append(":[indexSort=");
> s.append(indexSort);
> s.append(']');
>   }
>   // New Code
>   if (!diagnostics.isEmpty()) {
> s.append(":[diagnostics=");
> for (Map.Entry entry : diagnostics.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   // New Code
>   if (!attributes.isEmpty()) {
> s.append(":[attributes=");
> for (Map.Entry entry : attributes.entrySet())
>   
> s.append("<").append(entry.getKey()).append(",").append(entry.getValue()).append(">,");
> s.setLength(s.length() - 1);
> s.append(']');
>   }
>   return s.toString();
> }
> {code}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-http2-Linux (64bit/jdk-11) - Build # 8 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Linux/8/
Java: 64bit/jdk-11 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: 
[127.0.0.1:10218_solr, 127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr] Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/50)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   
"autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{   
"replicas":{"core_node1":{   
"core":"testCreateCollectionAddReplica_shard1_replica_n1",   
"SEARCHER.searcher.maxDoc":0,   "SEARCHER.searcher.deletedDocs":0,  
 "INDEX.sizeInBytes":10240,   "node_name":"127.0.0.1:10215_solr",   
"state":"active",   "type":"NRT",   
"INDEX.sizeInGB":9.5367431640625E-6,   "SEARCHER.searcher.numDocs":0}}, 
  "range":"8000-7fff",   "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10218_solr, 127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr]
Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/50)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
  "replicas":{"core_node1":{
  "core":"testCreateCollectionAddReplica_shard1_replica_n1",
  "SEARCHER.searcher.maxDoc":0,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":10240,
  "node_name":"127.0.0.1:10215_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":9.5367431640625E-6,
  "SEARCHER.searcher.numDocs":0}},
  "range":"8000-7fff",
  "state":"active"}}}
at 
__randomizedtesting.SeedInfo.seed([3A5771085D233A48:BA7714264C60D2EE]:0)
at 
org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 

[jira] [Updated] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Adrien Grand (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-7745:
-
Attachment: TermDisjunctionQuery.java

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: TermDisjunctionQuery.java, gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-7745) Explore GPU acceleration

2018-11-28 Thread Rinka Singh (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-7745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701855#comment-16701855
 ] 

Rinka Singh commented on LUCENE-7745:
-

Hi everyone,

I wanted to check if this issue was still open.  I have been experimenting with 
CUDA for a bit and would love to take a stab at this.

A few thoughts:
 * This is something I'll do over weekends and so I'm going to be horribly slow 
(its going to be just me on this unless you have someone working on it and I 
can collaborate with them) - would that be OK?
 * I think the right thing to do would be to build a CUDA library (C/C++), put 
JNI and then integrate it into Lucene.  If done right then I think this library 
will be useful to (and be possible to integrate with) other Analytic tools.
 * If I get it right, then I'd love to create an OS library that other OS tools 
can integrate and use (Yes, I'm thinking of an OpenCL port in the future but 
given the tools available in CUDA and my familiarity with it...)
 * Licensing is not an issue as I prefer the Apache License.
 * Testing (especially scalability testing) will be an issue - like you said, 
your setups won't have GPUs but would it be possible to rent a few GPU 
instances on the cloud (AWS, Google)?  I can do my dev testing locally as I 
have a GPU (its a  pretty old and obsolete one but good enough for my needs) on 
my dev machine.
 * It is important to get a few users who will experiment with this.  Can you 
guys help in having someone deploy, experiment and give feedback?
 * I would rather take something that is used by everyone and I'm thinking that 
indexing, filtering and searching is something that I would rather take up: 
[http://lucene.apache.org/core/7_5_0/demo/overview-summary.html#overview.description]
 ** These can certainly be accelerated.  I think I should be able to get some 
acceleration out of a GPU enabled search.
 ** The good part of this is one would able to scale volumes almost linearly on 
a multi-GPU machine.
 ** Related to the previous point (though this is in the future). I don't have 
a multi-GPU setup and will not be able to develop multi-GPU versions. I'll need 
help in getting the infrastructure to do that. We can talk about that once a 
single GPU version is done.
 ** Yes I agree that it will be better to have a separate library / classes 
doing this rather than directly integrating it into Lucene's class library.  
This suits me too as I can develop this as a separate library that other OS 
components can integrate and I can package this as part of nVidia's OS 
libraries.
 * I'm open to other alternatives - I scanned the ideas above but didn't 
consider them as they would not bring massive value to the users and I don't 
really want to experiment as I know what I'm doing.
 * Related to the previous point, I don't know Lucene (Help!! - do I really 
need to?) and will need support/hand-holding in terms of reviewing the 
identification/interfacing/design/code etc., etc.,
 * Finally, this IS GOING TO take time because thinking (and programming) 
massively parallel is completely different from writing a simple sequential 
search and sort.  How much time, think 7-10x at least given all my constraints.

If you guys like, I can write a brief (one or two paras) description of what is 
possible for indexing, searching, filtering (with zero knowledge of Lucene of 
course) to start off...

Your thoughts please...

> Explore GPU acceleration
> 
>
> Key: LUCENE-7745
> URL: https://issues.apache.org/jira/browse/LUCENE-7745
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: gsoc2017, mentor
> Attachments: gpu-benchmarks.png
>
>
> There are parts of Lucene that can potentially be speeded up if computations 
> were to be offloaded from CPU to the GPU(s). With commodity GPUs having as 
> high as 12GB of high bandwidth RAM, we might be able to leverage GPUs to 
> speed parts of Lucene (indexing, search).
> First that comes to mind is spatial filtering, which is traditionally known 
> to be a good candidate for GPU based speedup (esp. when complex polygons are 
> involved). In the past, Mike McCandless has mentioned that "both initial 
> indexing and merging are CPU/IO intensive, but they are very amenable to 
> soaking up the hardware's concurrency."
> I'm opening this issue as an exploratory task, suitable for a GSoC project. I 
> volunteer to mentor any GSoC student willing to work on this this summer.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-7.6-Windows (64bit/jdk-9.0.4) - Build # 11 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Windows/11/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

14 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:54008/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:54008/solr
at 
__randomizedtesting.SeedInfo.seed([76E266779B520A76:B7121FDBB602C0D1]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:54528/solr

Stack Trace:

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-11) - Build # 23272 - Still Unstable!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23272/
Java: 64bit/jdk-11 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

22 tests failed.
FAILED:  org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting

Error Message:
Error from server at http://127.0.0.1:37995/solr/collection1_shard2_replica_n2: 
Expected mime type application/octet-stream but got text/html.   
 
Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update  HTTP ERROR 
404 Problem accessing /solr/collection1_shard2_replica_n2/update. 
Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at http://127.0.0.1:37995/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([65342A0E53B97F51:A783166650F98F29]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testRouting(CloudSolrClientTest.java:238)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-Tests-master - Build # 2964 - Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/2964/

1 tests failed.
FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime

Error Message:
Error from server at 
https://127.0.0.1:43021/solr/collection1_shard2_replica_n2: Expected mime type 
application/octet-stream but got text/html.Error 404 
Can not find: /solr/collection1_shard2_replica_n2/update  
HTTP ERROR 404 Problem accessing 
/solr/collection1_shard2_replica_n2/update. Reason: Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605  
  

Stack Trace:
org.apache.solr.client.solrj.impl.CloudSolrClient$RouteException: Error from 
server at https://127.0.0.1:43021/solr/collection1_shard2_replica_n2: Expected 
mime type application/octet-stream but got text/html. 


Error 404 Can not find: 
/solr/collection1_shard2_replica_n2/update

HTTP ERROR 404
Problem accessing /solr/collection1_shard2_replica_n2/update. Reason:
Can not find: 
/solr/collection1_shard2_replica_n2/updatehttp://eclipse.org/jetty;>Powered by Jetty:// 9.4.11.v20180605




at 
__randomizedtesting.SeedInfo.seed([ABA5DA81DB4AC1B0:457DA11095351423]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:551)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1016)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:949)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testParallelUpdateQTime(CloudSolrClientTest.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[GitHub] lucene-solr issue #455: SOLR-12638

2018-11-28 Thread moshebla
Github user moshebla commented on the issue:

https://github.com/apache/lucene-solr/pull/455
  
I rebased using the latest master.
I also changed the tests so _root_ is not hard-coded into childless 
documents, since SOLR-5211 was committed to master and ensures this behaviour.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Toke Eskildsen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701727#comment-16701727
 ] 

Toke Eskildsen commented on SOLR-13013:
---

I cherry-picked some DENSE fields from our netarchive index and tried exporting 
them from a single shard, to demonstrate the problem with large indexes in 
Lucene/Solr 7+ and to performance test the current patch.

I made sure everything was warmed (practically zero IO on the index-SSD 
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374 
turned on and off:
{code}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504

> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq 
> .response.numFound'
52654

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
>  -o t_export_true_true
0.433661 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
>  -o t_export_true_false
0.555844 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
>  -o t_export_false_true
1.037004 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=false;
>  -o t_export_false_false
843.477925 seconds

> diff -s t_export_true_true t_export_true_false ; diff -s t_export_true_true 
> t_export_false_true ; diff -s t_export_true_true t_export_false_false
Files t_export_true_true and t_export_true_false are identical
Files t_export_true_true and t_export_false_true are identical
Files t_export_true_true and t_export_false_false are identical
{code}
Observations from this ad-hoc test (which of course should be independently 
verified): Exporting from a large index with vanilla Solr master is not ideal. 
It does not make much sense to talk about what performance-factors the patches 
provides as they are mostly about changing time complexity: Our factor 1500 
speed-up with SOLR-13013 with this shard with this request will be something 
quite else for other setups.
 * The explicit sort in SOLR-13013 seems the superior solution and the addition 
of the O(n) → O(1) lookup-improvement in LUCENE-8374 only makes it slightly 
faster.
 * On the other hand, LUCENE-8374 works quite well for export and does not 
require any changes to export. This might influence whether or not energy 
should be spend on a "best as possible" fallback in case of memory problems or 
if simpler "full fallback to sliding window sort order" is preferable.
 * On the gripping hand, testing with a smaller index is likely to result in 
SOLR-13013 being (relative to LUCENE-8374) even faster, as SOLR-13013 avoids 
re-opening DV-readers all the time. More testing needed (no surprise there).

> Change export to extract DocValues in docID order
> -
>
> Key: SOLR-13013
> URL: https://issues.apache.org/jira/browse/SOLR-13013
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Export Writer
>Affects Versions: 7.5, master (8.0)
>Reporter: Toke Eskildsen
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: SOLR-13013_proof_of_concept.patch, 
> SOLR-13013_proof_of_concept.patch
>
>
> The streaming export writer uses a sliding window of 30,000 documents for 
> paging through the result set in a given sort order. Each time a window has 
> been calculated, the values for the export fields are retrieved from the 
> underlying DocValues structures in document sort order and delivered.
> The iterative DocValues API introduced in Lucene/Solr 7 does not support 
> random access. The current export implementation bypasses this by creating a 
> new DocValues-iterator for each individual value to retrieve. This slows down 
> export as the iterator has to seek to the given docID from start for each 
> value. The slowdown scales with shard size (see LUCENE-8374 for details). An 
> alternative is to extract the DocValues in docID-order, with re-use of 
> DocValues-iterators. The idea is as follows:
>  # Change the FieldWriters for export to re-use the DocValues-iterators if 
> subsequent requests are for docIDs higher than the previous ones
>  # Calculate the sliding window of SortDocs as usual
>  # Take a note of the order of the SortDocs in the sliding window
>  # Re-sort the SortDocs in docID-order
>  # 

[jira] [Comment Edited] (SOLR-13013) Change export to extract DocValues in docID order

2018-11-28 Thread Toke Eskildsen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-13013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701727#comment-16701727
 ] 

Toke Eskildsen edited comment on SOLR-13013 at 11/28/18 11:15 AM:
--

I cherry-picked some DENSE fields from our netarchive index and tried exporting 
them from a single shard, to demonstrate the problem with large indexes in 
Lucene/Solr 7+ and to performance test the current patch.

I made sure everything was warmed (practically zero IO on the index-SSD 
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374 
turned on and off:
{code:java}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504

> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq 
> .response.numFound'
52654

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
>  -o t_export_true_true
0.433661 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
>  -o t_export_true_false
0.555844 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
>  -o t_export_false_true
1.037004 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=false;
>  -o t_export_false_false
843.477925 seconds

> diff -s t_export_true_true t_export_true_false ; diff -s t_export_true_true 
> t_export_false_true ; diff -s t_export_true_true t_export_false_false
Files t_export_true_true and t_export_true_false are identical
Files t_export_true_true and t_export_false_true are identical
Files t_export_true_true and t_export_false_false are identical
{code}
Observations from this ad-hoc test (which of course should be independently 
verified): Exporting from a large index with vanilla Solr master is not ideal. 
It does not make much sense to talk about what performance-factors the patches 
provides as they are mostly about changing time complexity: Our factor 1500 
speed-up with SOLR-13013 with this shard with this request will be something 
quite else for other setups.
 * The explicit sort in SOLR-13013 seems the superior solution and the addition 
of the O\(n) → O(1) lookup-improvement in LUCENE-8374 only makes it slightly 
faster.
 * On the other hand, LUCENE-8374 works quite well for export and does not 
require any changes to export. This might influence whether or not energy 
should be spend on a "best as possible" fallback in case of memory problems or 
if simpler "full fallback to sliding window sort order" is preferable.
 * On the gripping hand, testing with a smaller index is likely to result in 
SOLR-13013 being (relative to LUCENE-8374) even faster, as SOLR-13013 avoids 
re-opening DV-readers all the time. More testing needed (no surprise there).


was (Author: toke):
I cherry-picked some DENSE fields from our netarchive index and tried exporting 
them from a single shard, to demonstrate the problem with large indexes in 
Lucene/Solr 7+ and to performance test the current patch.

I made sure everything was warmed (practically zero IO on the index-SSD 
according to iostat) and tested with combinations of SOLR-13013 and LUCENE-8374 
turned on and off:
{code}
> curl -s "http://localhost:9090/solr/ns80/select?q=*:*; | jq .response.numFound
307171504

> curl -s "http://localhost:9090/solr/ns80/select?q=text:hestevogn; | jq 
> .response.numFound'
52654

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=true;
>  -o t_export_true_true
0.433661 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=true=false;
>  -o t_export_true_false
0.555844 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=true;
>  -o t_export_false_true
1.037004 seconds

> curl -s -w "%\{time_total} seconds"$\'\n\' 
> "http://localhost:9090/solr/ns80/export?q=text:hestevogn=id+asc=content_type_ext,content_type_served,crawl_date,content_length=false=false;
>  -o t_export_false_false
843.477925 seconds

> diff -s t_export_true_true t_export_true_false ; diff -s t_export_true_true 
> t_export_false_true ; diff -s t_export_true_true t_export_false_false
Files 

[JENKINS] Lucene-Solr-http2-Windows (32bit/jdk1.8.0_172) - Build # 4 - Still Failing!

2018-11-28 Thread Policeman Jenkins Server
Build: https://jenkins.thetaphi.de/job/Lucene-Solr-http2-Windows/4/
Java: 32bit/jdk1.8.0_172 -client -XX:+UseParallelGC

4 tests failed.
FAILED:  
org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud

Error Message:
IOException occured when talking to server at: https://127.0.0.1:58203/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occured when 
talking to server at: https://127.0.0.1:58203/solr
at 
__randomizedtesting.SeedInfo.seed([5CAEDFA0CB32BFBF:8DA92D256F3D348D]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:657)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:367)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:295)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:213)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:196)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:213)
at 
org.apache.solr.cloud.LegacyCloudClusterPropTest.createAndTest(LegacyCloudClusterPropTest.java:80)
at 
org.apache.solr.cloud.LegacyCloudClusterPropTest.testCreateCollectionSwitchLegacyCloud(LegacyCloudClusterPropTest.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

RE: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Uwe Schindler
Hi,

 

About the bundling of conscrypt.jar in the binary distribution of Apache Solr, 
I opened LEGAL-425:

https://issues.apache.org/jira/browse/LEGAL-425

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de

 

From: Uwe Schindler  
Sent: Wednesday, November 28, 2018 11:38 AM
To: dev@lucene.apache.org
Subject: RE: Poll: Merge jira/http2 to master branch

 

Hi Dat, hi Shawn,

 

Thanks for the confirmation! This is what I had in mind (missing ALPN support).

 

IMHO, we should not yet switch away from Java 8 as minimum requirement. With 
multi-release JARs we are at the moment perfectly able to handle users with 
newer Java versions, but we should still support Java 8 (as its still widely 
deployed), although EOL is close. As Redhat, AdoptOpenJDK, and several Linux 
distributions still provide support for Java 8 at least for 2 or 3 more years, 
I think it’s to early to switch. With recent changes, the support by Oracle is 
no longer the way to go, as there are many alternatives.

 

My proposal would be to wait until master is branched to “branch_8x” (stays on 
Java 8 + MR-JAR support), then change “master” to have Java 11 as minimum 
requirement.

 

About HTTP2: I have no problem with bundling conscrypt (is it needed for both 
Jetty Server and Jetty Client)?, but disable it by default and only register it 
in the Java TLS SPI, if Java 8 is used. On Java 9 and later use the shipped 
HTTP2 support. My proposal would be to do it with a Multirelease JAR (this is 
currently enabled in Lucene already).

 

If there are license problems, we can do the following: Disable HTTP2 on Java 8 
completely but provide documentation in the Solr Ref Guide how to enable it 
(something like: download conscrypt-uber.jar from maven and save in solr/lib 
directory). We can enable it automatically, if class.forName() does not throw 
CNFE). Maybe add a sysprop, but that’s not needed. I think enabling conscrypt 
is easy with reflection only, or do we need to compile hard against it. From 
what I figured out, it’s only used at a few places.

 

SolrJ should by default ship without conscyrpt (make it “optional” maven 
dependency). If it’s in classpath, enable it.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de  

 

From: Đạt Cao Mạnh mailto:caomanhdat...@gmail.com> > 
Sent: Wednesday, November 28, 2018 11:01 AM
To: Solr/Lucene Dev mailto:dev@lucene.apache.org> >
Subject: Re: Poll: Merge jira/http2 to master branch

 

Hi,

 

I will try to summary all the options here

1. No support for HTTP/2 + SSL at all, if users want to run SSL they must stick 
with HTTP/1.1

2. Set Java requirement to 9 and use ALPN implementation of JDK 9

3. If users want to use HTTP/2, we will notice about license problem then 
download and use Conscrypt library. Of course that we still test the Conscrypt 
implementation in tests.

 

 

On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey mailto:apa...@elyograg.org> > wrote:

On 11/27/2018 3:32 AM, Uwe Schindler wrote:
> I'd like to bring one thing into attention: This library conscrypt is 
> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork 
> named BoringSSL. I'd recommend to check the licensing, because OpenSSL 
> licenses are a bit strange (some BSD fork, also ASF2, but some code is still 
> outdated - I am not sure what the fork actually uses). I have the feeling we 
> should include ASF legal department.
>
> Nevertheless, I am not 100% sure if conscrypt should really be inclued by 
> default in SolrJ. As SolrJ is a client library for Solr and can be used by 
> external projects, there is the problem of incompatibilities with the native 
> C code included. E.g., if one uses SolrJ with IBM J9 or other Java versions 
> different from openjdk. So I'd be careful. My question is: Do we really need 
> that library. To me it looks like it improves speed only, or is there 
> something that requires its use?

As you might know ... full and proper http/2 support with Java 8 
requires the conscrypt library.  With Java 9 or higher, the 
functionality is provided natively by Java.  If I remember right, what 
conscrypt provides that Java 8 lacks is ALPN.

If there are license issues with conscrypt, perhaps it might make sense 
to require a newer major version of Java instead ... but I think that 
upgrading the project's Java requirement to 9, 10, or 11 probably 
requires a wider discussion.  Lucene probably has no burning need for it 
right now.  I have no idea whether there are language features in Java 
9+ that we as a group are wanting to use.  Another point for 
consideration is that the effective EOL for Java 8 is fast approaching, 
and EOL dates have already come and gone for 9 and 10.

Thanks,
Shawn


-
To unsubscribe, 

RE: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Uwe Schindler
Hi Dat, hi Shawn,

 

Thanks for the confirmation! This is what I had in mind (missing ALPN support).

 

IMHO, we should not yet switch away from Java 8 as minimum requirement. With 
multi-release JARs we are at the moment perfectly able to handle users with 
newer Java versions, but we should still support Java 8 (as its still widely 
deployed), although EOL is close. As Redhat, AdoptOpenJDK, and several Linux 
distributions still provide support for Java 8 at least for 2 or 3 more years, 
I think it’s to early to switch. With recent changes, the support by Oracle is 
no longer the way to go, as there are many alternatives.

 

My proposal would be to wait until master is branched to “branch_8x” (stays on 
Java 8 + MR-JAR support), then change “master” to have Java 11 as minimum 
requirement.

 

About HTTP2: I have no problem with bundling conscrypt (is it needed for both 
Jetty Server and Jetty Client)?, but disable it by default and only register it 
in the Java TLS SPI, if Java 8 is used. On Java 9 and later use the shipped 
HTTP2 support. My proposal would be to do it with a Multirelease JAR (this is 
currently enabled in Lucene already).

 

If there are license problems, we can do the following: Disable HTTP2 on Java 8 
completely but provide documentation in the Solr Ref Guide how to enable it 
(something like: download conscrypt-uber.jar from maven and save in solr/lib 
directory). We can enable it automatically, if class.forName() does not throw 
CNFE). Maybe add a sysprop, but that’s not needed. I think enabling conscrypt 
is easy with reflection only, or do we need to compile hard against it. From 
what I figured out, it’s only used at a few places.

 

SolrJ should by default ship without conscyrpt (make it “optional” maven 
dependency). If it’s in classpath, enable it.

 

Uwe

 

-

Uwe Schindler

Achterdiek 19, D-28357 Bremen

http://www.thetaphi.de  

eMail: u...@thetaphi.de

 

From: Đạt Cao Mạnh  
Sent: Wednesday, November 28, 2018 11:01 AM
To: Solr/Lucene Dev 
Subject: Re: Poll: Merge jira/http2 to master branch

 

Hi,

 

I will try to summary all the options here

1. No support for HTTP/2 + SSL at all, if users want to run SSL they must stick 
with HTTP/1.1

2. Set Java requirement to 9 and use ALPN implementation of JDK 9

3. If users want to use HTTP/2, we will notice about license problem then 
download and use Conscrypt library. Of course that we still test the Conscrypt 
implementation in tests.

 

 

On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey mailto:apa...@elyograg.org> > wrote:

On 11/27/2018 3:32 AM, Uwe Schindler wrote:
> I'd like to bring one thing into attention: This library conscrypt is 
> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork 
> named BoringSSL. I'd recommend to check the licensing, because OpenSSL 
> licenses are a bit strange (some BSD fork, also ASF2, but some code is still 
> outdated - I am not sure what the fork actually uses). I have the feeling we 
> should include ASF legal department.
>
> Nevertheless, I am not 100% sure if conscrypt should really be inclued by 
> default in SolrJ. As SolrJ is a client library for Solr and can be used by 
> external projects, there is the problem of incompatibilities with the native 
> C code included. E.g., if one uses SolrJ with IBM J9 or other Java versions 
> different from openjdk. So I'd be careful. My question is: Do we really need 
> that library. To me it looks like it improves speed only, or is there 
> something that requires its use?

As you might know ... full and proper http/2 support with Java 8 
requires the conscrypt library.  With Java 9 or higher, the 
functionality is provided natively by Java.  If I remember right, what 
conscrypt provides that Java 8 lacks is ALPN.

If there are license issues with conscrypt, perhaps it might make sense 
to require a newer major version of Java instead ... but I think that 
upgrading the project's Java requirement to 9, 10, or 11 probably 
requires a wider discussion.  Lucene probably has no burning need for it 
right now.  I have no idea whether there are language features in Java 
9+ that we as a group are wanting to use.  Another point for 
consideration is that the effective EOL for Java 8 is fast approaching, 
and EOL dates have already come and gone for 9 and 10.

Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org 
 




 

-- 

Best regards,

Cao Mạnh Đạt

D.O.B : 31-07-1991
Cell: (+84) 946.328.329
E-mail:   caomanhdat...@gmail.com



Re: Poll: Merge jira/http2 to master branch

2018-11-28 Thread Đạt Cao Mạnh
Hi,

I will try to summary all the options here
1. No support for HTTP/2 + SSL at all, if users want to run SSL they must
stick with HTTP/1.1
2. Set Java requirement to 9 and use ALPN implementation of JDK 9
3. If users want to use HTTP/2, we will notice about license problem then
download and use Conscrypt library. Of course that we still test the
Conscrypt implementation in tests.


On Wed, Nov 28, 2018 at 1:06 AM Shawn Heisey  wrote:

> On 11/27/2018 3:32 AM, Uwe Schindler wrote:
> > I'd like to bring one thing into attention: This library conscrypt is
> ASF2-licensed, but the JAR files contain binary versions of an OpenSSL fork
> named BoringSSL. I'd recommend to check the licensing, because OpenSSL
> licenses are a bit strange (some BSD fork, also ASF2, but some code is
> still outdated - I am not sure what the fork actually uses). I have the
> feeling we should include ASF legal department.
> >
> > Nevertheless, I am not 100% sure if conscrypt should really be inclued
> by default in SolrJ. As SolrJ is a client library for Solr and can be used
> by external projects, there is the problem of incompatibilities with the
> native C code included. E.g., if one uses SolrJ with IBM J9 or other Java
> versions different from openjdk. So I'd be careful. My question is: Do we
> really need that library. To me it looks like it improves speed only, or is
> there something that requires its use?
>
> As you might know ... full and proper http/2 support with Java 8
> requires the conscrypt library.  With Java 9 or higher, the
> functionality is provided natively by Java.  If I remember right, what
> conscrypt provides that Java 8 lacks is ALPN.
>
> If there are license issues with conscrypt, perhaps it might make sense
> to require a newer major version of Java instead ... but I think that
> upgrading the project's Java requirement to 9, 10, or 11 probably
> requires a wider discussion.  Lucene probably has no burning need for it
> right now.  I have no idea whether there are language features in Java
> 9+ that we as a group are wanting to use.  Another point for
> consideration is that the effective EOL for Java 8 is fast approaching,
> and EOL dates have already come and gone for 9 and 10.
>
> Thanks,
> Shawn
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

-- 
*Best regards,*
*Cao Mạnh Đạt*


*D.O.B : 31-07-1991Cell: (+84) 946.328.329E-mail: caomanhdat...@gmail.com
*


[jira] [Resolved] (LUCENE-8573) BKDWriter should use FutureArrays#mismatch

2018-11-28 Thread Adrien Grand (JIRA)


 [ 
https://issues.apache.org/jira/browse/LUCENE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8573.
--
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

Thanks [~cbuescher]!

> BKDWriter should use FutureArrays#mismatch
> --
>
> Key: LUCENE-8573
> URL: https://issues.apache.org/jira/browse/LUCENE-8573
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In a number of places, BKDWriter tries to find the first mismatching byte 
> between multiple arrays with a for loop. It could use the safer 
> FutureArrays#mismatch instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[GitHub] lucene-solr pull request #510: LUCENE-8573: Use FutureArrays#mismatch in BKD...

2018-11-28 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/lucene-solr/pull/510


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8573) BKDWriter should use FutureArrays#mismatch

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701621#comment-16701621
 ] 

ASF subversion and git services commented on LUCENE-8573:
-

Commit ecce083c5017576f552a5fd8e347b8748dbc0db9 in lucene-solr's branch 
refs/heads/master from [~cbuescher]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ecce083 ]

LUCENE-8573: Use FutureArrays#mismatch in BKDWriter

Closes #510

Signed-off-by: Adrien Grand 


> BKDWriter should use FutureArrays#mismatch
> --
>
> Key: LUCENE-8573
> URL: https://issues.apache.org/jira/browse/LUCENE-8573
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In a number of places, BKDWriter tries to find the first mismatching byte 
> between multiple arrays with a for loop. It could use the safer 
> FutureArrays#mismatch instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Commented] (LUCENE-8573) BKDWriter should use FutureArrays#mismatch

2018-11-28 Thread ASF subversion and git services (JIRA)


[ 
https://issues.apache.org/jira/browse/LUCENE-8573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16701620#comment-16701620
 ] 

ASF subversion and git services commented on LUCENE-8573:
-

Commit cf92418711dfe16862b66f2c14e176ac1697d3fc in lucene-solr's branch 
refs/heads/branch_7x from [~cbuescher]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=cf92418 ]

LUCENE-8573: Use FutureArrays#mismatch in BKDWriter

Closes #510

Signed-off-by: Adrien Grand 


> BKDWriter should use FutureArrays#mismatch
> --
>
> Key: LUCENE-8573
> URL: https://issues.apache.org/jira/browse/LUCENE-8573
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: master (8.0), 7.7
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> In a number of places, BKDWriter tries to find the first mismatching byte 
> between multiple arrays with a for loop. It could use the safer 
> FutureArrays#mismatch instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[jira] [Created] (LUCENE-8577) Improve accuracy of distance query for Geo3DPoint

2018-11-28 Thread Ignacio Vera (JIRA)
Ignacio Vera created LUCENE-8577:


 Summary: Improve accuracy of distance query for Geo3DPoint
 Key: LUCENE-8577
 URL: https://issues.apache.org/jira/browse/LUCENE-8577
 Project: Lucene - Core
  Issue Type: Improvement
  Components: modules/spatial3d
Reporter: Ignacio Vera


The distance query for Geo3DPoint currently uses the GeoStandardCircle which is 
only accurate when the PlanetModel is a sphere. 

In LUCENE-7970. a new implementation called GeoExactCircle was added that 
models an exact circle, even when the planet model is not a sphere. Because 
Geo3DPoint is using WGS84 planet model,  we should be using this implementation 
instead. The downside is that this implementation will be slightly slower. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[GitHub] lucene-solr issue #510: LUCENE-8573: Use FutureArrays#mismatch in BKDWriter

2018-11-28 Thread uschindler
Github user uschindler commented on the issue:

https://github.com/apache/lucene-solr/pull/510
  
Perfect. So it's even more performant with Java 9+.
Thanks, Christian


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org



[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1708 - Still Unstable

2018-11-28 Thread Apache Jenkins Server
Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1708/

5 tests failed.
FAILED:  org.apache.solr.cloud.RestartWhileUpdatingTest.test

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([813A3D1CE93389BC]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.RestartWhileUpdatingTest

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([813A3D1CE93389BC]:0)


FAILED:  org.apache.solr.cloud.TestRebalanceLeaders.test

Error Message:
Error from server at https://127.0.0.1:36779/_l/qc: KeeperErrorCode = Session 
expired for /configs/conf1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:36779/_l/qc: KeeperErrorCode = Session expired 
for /configs/conf1
at 
__randomizedtesting.SeedInfo.seed([813A3D1CE93389BC:96E02C647CFE444]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1107)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1676)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1703)
at 
org.apache.solr.cloud.TestRebalanceLeaders.test(TestRebalanceLeaders.java:71)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

  1   2   >