[JENKINS] Lucene-Solr-BadApples-Tests-7.x - Build # 221 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-7.x/221/

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest.testInactiveShardCleanup

Error Message:
missing cleanup event: [CapturedEvent{timestamp=3959519998854257, 
stage=STARTED, actionName='null', event={   
"id":"e1129967f2d9eT31vbcageilp1e05yc128hs4r2",   
"source":".scheduled_maintenance",   "eventTime":3959519990197662,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824737030,  
   "_enqueue_time_":3959519995294017}}, context={}, config={   
"trigger":".scheduled_maintenance",   "stage":[ "STARTED", "ABORTED",   
  "SUCCEEDED", "FAILED"],   "beforeAction":"inactive_shard_plan",   
"afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959520016694334, 
stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   
"id":"e1129967f2d9eT31vbcageilp1e05yc128hs4r2",   
"source":".scheduled_maintenance",   "eventTime":3959519990197662,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824737030,  
   "_enqueue_time_":3959519995294017}}, 
context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], 
source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",  
 "stage":[ "STARTED", "ABORTED", "SUCCEEDED", "FAILED"],   
"beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959520028293027, stage=AFTER_ACTION, 
actionName='inactive_shard_plan', event={   
"id":"e1129967f2d9eT31vbcageilp1e05yc128hs4r2",   
"source":".scheduled_maintenance",   "eventTime":3959519990197662,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824737030,  
   "_enqueue_time_":3959519995294017}}, 
context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], 
source=.scheduled_maintenance, 
properties.inactive_shard_plan={staleLocks={ScheduledMaintenanceTriggerTest_collection1/staleShard-splitting={stateTimestamp=1542651936989931990,
 currentTimeNs=1542824737068663834, deltaSec=172800, ttlSec=20}}}, 
properties.AFTER_ACTION=[inactive_shard_plan, execute_plan, test]}, config={   
"trigger":".scheduled_maintenance",   "stage":[ "STARTED", "ABORTED",   
  "SUCCEEDED", "FAILED"],   "beforeAction":"inactive_shard_plan",   
"afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959520030815134, stage=SUCCEEDED, 
actionName='null', event={   "id":"e1129967f2d9eT31vbcageilp1e05yc128hs4r2",   
"source":".scheduled_maintenance",   "eventTime":3959519990197662,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824737030,  
   "_enqueue_time_":3959519995294017}}, context={}, config={   
"trigger":".scheduled_maintenance",   "stage":[ "STARTED", "ABORTED",   
  "SUCCEEDED", "FAILED"],   "beforeAction":"inactive_shard_plan",   
"afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959525035509030, stage=STARTED, 
actionName='null', event={   "id":"e112ac3260f7dT31vbcageilp1e05yc128hs4r8",   
"source":".scheduled_maintenance",   "eventTime":3959525034299261,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824742075,  
   "_enqueue_time_":3959525034696618}}, context={}, config={   
"trigger":".scheduled_maintenance",   "stage":[ "STARTED", "ABORTED",   
  "SUCCEEDED", "FAILED"],   "beforeAction":"inactive_shard_plan",   
"afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959525036370014, 
stage=BEFORE_ACTION, actionName='inactive_shard_plan', event={   
"id":"e112ac3260f7dT31vbcageilp1e05yc128hs4r8",   
"source":".scheduled_maintenance",   "eventTime":3959525034299261,   
"eventType":"SCHEDULED",   "properties":{ "actualEventTime":1542824742075,  
   "_enqueue_time_":3959525034696618}}, 
context={properties.BEFORE_ACTION=[inactive_shard_plan, execute_plan, test], 
source=.scheduled_maintenance}, config={   "trigger":".scheduled_maintenance",  
 "stage":[ "STARTED", "ABORTED", "SUCCEEDED", "FAILED"],   
"beforeAction":"inactive_shard_plan",   "afterAction":"inactive_shard_plan",   
"class":"org.apache.solr.cloud.autoscaling.ScheduledMaintenanceTriggerTest$CapturingTriggerListener"},
 message='null'}, CapturedEvent{timestamp=3959525040124348, stage=AFTER_ACTION, 
actionName='inactive_shard_plan', event={   

Re: SOLR: Unnecessary logging

[Uwe Schindler]

No it won't deduplicate. It is actually worse: Every class would be a different 
class instance per core, just with same name (they won't even compare equals to 
each other). This why you have classloader, so you can have same class name in 
different parts of app. OSGI... The expressions module does the same. Every 
compiled Lucene expression has the same class name, just in a different 
classloader.

So every core has its own copy of all classes. That's a common problem for 
users with many cores each of those having Tika configured per core only...

Uwe

Am November 21, 2018 10:23:29 PM UTC schrieb Shawn Heisey :
>On 11/21/2018 1:40 AM, Jan Høydahl wrote:
>> It is super easy to change the log level of one particular class or 
>> package
>> selectively, that's why we have log4j2.xml and also the Admin's 
>> Logging levels menu.
>> The occational need to see all of the 150 jar files loaded on startup
>
>> should not be
>> solved by spewing out that for first-time users of Solr who really 
>> just care for whether
>> the server started OK or not.
>
>If somebody's got a config that loads 150 jars, I really do think that 
>having that info in the log would be useful information.  That's a LOT 
>of jars, and it could be vital for troubleshooting a classloader
>problem 
>with one jar -- need to be able to tell whether it has been loaded or
>not.
>
>It definitely shouldn't be default to have Jetty log its jar loading. 
>I 
>just want to know how to make it do so, for situations that require
>very 
>in-depth study.
>
>A user who's got 20 cores and each of them is loading the same 10 jars 
>with  config ... IMHO that would ALSO be useful information to be 
>able to see in the default log.  Without it, the user will probably
>have 
>absolutely no idea that they have 200 jar loads.  (when that happens, 
>does Java realize that the classloader is loading the same jars and 
>eliminate duplicates in memory?)
>
>Thanks,
>Shawn
>
>
>-
>To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>For additional commands, e-mail: dev-h...@lucene.apache.org

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de

[jira] [Commented] (SOLR-9856) Collect metrics for shard replication and tlog replay on replicas

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695245#comment-16695245
 ] 

ASF subversion and git services commented on SOLR-9856:
---

Commit a31ac7f01121dc4d6fd02104f444a7224912aa89 in lucene-solr's branch 
refs/heads/branch_7_6 from Andrzej Bialecki
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a31ac7f ]

SOLR-9856: Update the Ref Guide to no longer list this issue as unresolved.


> Collect metrics for shard replication and tlog replay on replicas
> -
>
> Key: SOLR-9856
> URL: https://issues.apache.org/jira/browse/SOLR-9856
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.0
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: 6.4, 7.0
>
> Attachments: SOLR-9856.patch
>
>
> Using API from SOLR-4735 add metrics for tracking outgoing replication from 
> leader to shard replicas, and for tracking transaction log processing on 
> replicas.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9856) Collect metrics for shard replication and tlog replay on replicas

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695246#comment-16695246
 ] 

Andrzej Bialecki  commented on SOLR-9856:
-

Fixed - thanks Christine!

> Collect metrics for shard replication and tlog replay on replicas
> -
>
> Key: SOLR-9856
> URL: https://issues.apache.org/jira/browse/SOLR-9856
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.0
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: 6.4, 7.0
>
> Attachments: SOLR-9856.patch
>
>
> Using API from SOLR-4735 add metrics for tracking outgoing replication from 
> leader to shard replicas, and for tracking transaction log processing on 
> replicas.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-12-ea+12) - Build # 23245 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23245/
Java: 64bit/jdk-12-ea+12 -XX:+UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.DocValuesNotIndexedTest

Error Message:
Collection not found: dv_coll

Stack Trace:
org.apache.solr.common.SolrException: Collection not found: dv_coll
at __randomizedtesting.SeedInfo.seed([92E5E2EEC736CFB3]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:851)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.cloud.DocValuesNotIndexedTest.createCluster(DocValuesNotIndexedTest.java:154)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest

Error Message:
Could not find collection : AutoscalingHistoryHandlerTest_collection

Stack Trace:
org.apache.solr.common.SolrException: Could not find collection : 
AutoscalingHistoryHandlerTest_collection
at __randomizedtesting.SeedInfo.seed([92E5E2EEC736CFB3]:0)
at 
org.apache.solr.common.cloud.ClusterState.getCollection(ClusterState.java:118)
at 
org.apache.solr.cloud.SolrCloudTestCase.getCollectionState(SolrCloudTestCase.java:258)
at 
org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.waitForRecovery(AutoscalingHistoryHandlerTest.java:403)
at 
org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest.setupCluster(AutoscalingHistoryHandlerTest.java:97)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 36 - Still unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/36/

9 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:40019/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:40019/collection1
at 
__randomizedtesting.SeedInfo.seed([57FF9BDCA1BA41A:8D2BC66764E7C9E2]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:509)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:498)
at 
org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

Re: Lucene/Solr 7.6

[Cassandra Targett]

Doc changes are still fine, Andrzej. I still have a couple things to do for
the Ref Guide also.

On Wed, Nov 21, 2018 at 12:09 PM Andrzej Białecki  wrote:

> Hi Nicholas,
>
> If it’s ok I would like to merge a small fix to the Ref Guide, spotted by
> Christine in SOLR-9856.
>
> On 1 Nov 2018, at 21:38, Nicholas Knize  wrote:
>
> Hi all,
>
> To follow up from our discussion on the 8.0 thread, I would like to cut
> the 7.6 branch on either Tuesday or Wednesday of next week. Since this
> implies feature freeze I went ahead and had a look at some of the issues
> that are labeled for 7.6.
>
> It looks like we only have one active issue listed as a blocker for Solr.
> The upgrade notes in SOLR-12927
> 
>
> For Lucene we have five active issues (each with a patch provided) listed
> as blockers targeted for 7.6.
>
> If there are any other issues that need to land before cutting the branch,
> and they are not already labeled, please either mark them as blockers
> accordingly or let me know prior to cutting the branch next Tuesday or
> Wednesday.
>
> Thank you!
>
> - Nick
> --
>
> Nicholas Knize, Ph.D., GISP
> Geospatial Software Guy  |  Elasticsearch
> Apache Lucene Committer
> nkn...@apache.org
>
>
>
>
> —
>
> Andrzej Białecki
>
>

[JENKINS] Lucene-Solr-7.6-Linux (32bit/jdk1.8.0_172) - Build # 12 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/12/
Java: 32bit/jdk1.8.0_172 -client -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: 
[127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 127.0.0.1:10213_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr] Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/32)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   
"autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{   
"replicas":{"core_node1":{   
"core":"testCreateCollectionAddReplica_shard1_replica_n1",   
"SEARCHER.searcher.maxDoc":0,   "SEARCHER.searcher.deletedDocs":0,  
 "INDEX.sizeInBytes":10240,   "node_name":"127.0.0.1:10213_solr",   
"state":"active",   "type":"NRT",   
"INDEX.sizeInGB":9.5367431640625E-6,   "SEARCHER.searcher.numDocs":0}}, 
  "range":"8000-7fff",   "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10214_solr, 127.0.0.1:10216_solr, 127.0.0.1:10213_solr, 
127.0.0.1:10217_solr, 127.0.0.1:10215_solr]
Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/32)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
  "replicas":{"core_node1":{
  "core":"testCreateCollectionAddReplica_shard1_replica_n1",
  "SEARCHER.searcher.maxDoc":0,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":10240,
  "node_name":"127.0.0.1:10213_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":9.5367431640625E-6,
  "SEARCHER.searcher.numDocs":0}},
  "range":"8000-7fff",
  "state":"active"}}}
at 
__randomizedtesting.SeedInfo.seed([BCF4BBDA16D3BA09:3CD4DEF4079052AF]:0)
at 
org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 

[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter

[Cetra Free (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695315#comment-16695315
 ] 

Cetra Free commented on SOLR-13003:
---

With regards to {{FastLRUCache}}: the actual accounting implementation is in 
{{ConcurrentLRUCache}} which appears to be doing the right thing.  But the 
issue is the size comparison, so I don't think swapping would help. I will give 
it a shot regardless.

I've noticed the QueryResultKey is greater than 75kb of memory per key, which 
is much greater size than the value.

I will try out the patched version from Shawn & see if I can trigger it and get 
some stats out.

> Query Result Cache does not honour maxRamBytes parameter
> 
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: CLRU-logging.patch, lrucacheexpanded.png, 
> lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we 
> have seen the retained size of the cache orders of magnitude larger than what 
> is configured.
> Please see attached VisualVM output which shows the retained size is about 
> 1.5gb, but the maxRamBytes is set to 64mb.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Shawn Heisey]

On 11/21/2018 1:40 AM, Jan Høydahl wrote:
It is super easy to change the log level of one particular class or 
package
selectively, that's why we have log4j2.xml and also the Admin's 
Logging levels menu.
The occational need to see all of the 150 jar files loaded on startup 
should not be
solved by spewing out that for first-time users of Solr who really 
just care for whether

the server started OK or not.


If somebody's got a config that loads 150 jars, I really do think that 
having that info in the log would be useful information.  That's a LOT 
of jars, and it could be vital for troubleshooting a classloader problem 
with one jar -- need to be able to tell whether it has been loaded or not.


It definitely shouldn't be default to have Jetty log its jar loading.  I 
just want to know how to make it do so, for situations that require very 
in-depth study.


A user who's got 20 cores and each of them is loading the same 10 jars 
with  config ... IMHO that would ALSO be useful information to be 
able to see in the default log.  Without it, the user will probably have 
absolutely no idea that they have 200 jar loads.  (when that happens, 
does Java realize that the classloader is loading the same jars and 
eliminate duplicates in memory?)


Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13002) running solr in docker and tomcat in docker

[Gus Heck (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695293#comment-16695293
 ] 

Gus Heck commented on SOLR-13002:
-

This should be raised on the user list. Also please be aware that no recent 
version of solr supports deployment in tomcat, nor is tomcat particular to solr 
features, so there are almost no legitimate reasons to mention tomcat in a solr 
issue anymore. This tracker is for solr only

> running solr in docker and tomcat in docker
> ---
>
> Key: SOLR-13002
> URL: https://issues.apache.org/jira/browse/SOLR-13002
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: config-api
>Affects Versions: 7.4
> Environment: host with 64 GB Ram 6 cores running about 20 development 
> containers in docker ce 18.06.1 
>Reporter: Ray Ort
>Priority: Major
>  Labels: newbie
>
> I have been trying to connect a dockerized instance of solr to a dockerized 
> instance of tomcat
> both servers come up, solr indexes the files shared with tomcat connected to 
> the host tomcat should provide tiles from the search requested search in an 
> httpd server (that is also dockerized) which handles makes the request to 
> solr and get the result from tomcat in the form of text and images.  The logs 
> all three show that the servers come up a search of the core returns data but 
>  a curl returns a 404 error on the solr server  returns following
> Solr process 1 running on port 8983
> INFO - 2018-11-20 17:23:22.572; 
> org.apache.solr.util.configuration.SSLCredentialProviderFactory; Processing 
> SSL Credential Provider chain: env;sysprop
> {
>  "solr_home":"/opt/solr/server/solr",
>  "version":"7.4.0 9060ac689c270b02143f375de0348b7f626adebc - jpountz - 
> 2018-06-18 16:55:13",
>  "startTime":"2018-11-20T15:51:27.013Z",
>  "uptime":"0 days, 1 hours, 31 minutes, 55 seconds",
>  "memory":"95.4 MB (%19.4) of 490.7 MB"}
>  
>  
> But using the url to test 
> [http://healthaccounts-solr.boa.com/solr/healthaccounts/select?q=sd500=json]
>   I get 
> {
>  "responseHeader":{
>  "status":0,
>  "QTime":0,
>  "params":{
>  "q":"sd500",
>  "wt":"json"}},
>  "response":{"numFound":0,"start":0,"docs":[]
>  }}
> nothing which is what the tomcat returns, not sure what I am missing on this 
> connection there is no ssl the path for my healthaccounts data is shared 
> between all 3 services tomcat. solr and apache, all the servers ping and solr 
> should be providing a search but none seems forthcoming lost in this for 3 
> plus weeks
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #503: LUCENE-8571: Don't block on FrozenBufferedUpd...

[mikemccand]

Github user mikemccand commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/503#discussion_r235350971
  
--- Diff: 
lucene/core/src/java/org/apache/lucene/index/FrozenBufferedUpdates.java ---
@@ -214,150 +216,174 @@ public FrozenBufferedUpdates(InfoStream infoStream, 
BufferedUpdates updates, Seg
 
   /** Translates a frozen packet of delete term/query, or doc values
*  updates, into their actual docIDs in the index, and applies the 
change.  This is a heavy
-   *  operation and is done concurrently by incoming indexing threads. */
+   *  operation and is done concurrently by incoming indexing threads.
+   *  This method will return immediately without blocking if another 
thread is currently
+   *  applying the package. In order to ensure the packet has been 
applied, {@link #forceApply(IndexWriter)}
+   *  must be called.
+   *  */
   @SuppressWarnings("try")
-  public synchronized void apply(IndexWriter writer) throws IOException {
-if (applied.getCount() == 0) {
-  // already done
-  return;
+  boolean tryApply(IndexWriter writer) throws IOException {
+if (applyLock.tryLock()) {
+  try {
+return forceApply(writer);
+  } finally {
+applyLock.unlock();
+  }
 }
+return false;
+  }
 
-long startNS = System.nanoTime();
+  /** Translates a frozen packet of delete term/query, or doc values
+   *  updates, into their actual docIDs in the index, and applies the 
change.  This is a heavy
+   *  operation and is done concurrently by incoming indexing threads.
+   *  */
+  boolean forceApply(IndexWriter writer) throws IOException {
--- End diff --

Can we make this `void`?  It always just `return true` now right?


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SynonymQuery / Query Expansion Strategies Discussion

[Alessandro Benedetti]

Hi all,
last sunday we spent a bit on this topic, our considerations follow:

N.B. we didn't check the state of the art (thanks Doug for the nice survey
shared, I will definitely take a look later on) .
So we just wanted to figure out an initial improvement, that can later be
advanced following advanced state of the art formulas.
It is kinda related to Jim idea.
This was the output of our brainstorming:

*Introduction*
Currently in Apache Solr (and Elastic Search) there is no supported way to
manage true synonyms, hypernyms and hyponyms at query time.
A first attempt to add the support for that was done by Doug Turnbull with
the approach in the following Pull Requests [1].
We think that approach was a good starting point, but we do believe it
could be improved.

*Weaknesses of Current Approach*
The current approach in our opinion presents the following weaknesses :
- try to guess the hypernym/hyponym/synonym relation from the DF of the
terms
- doesn't favour the original query term necessary
- favour rarer hypernym/hyponym/synonym and don't differentiate them.

*Proposed Improvements*

   - Nym Class Priority Order
   - Nyms within a Class Ranked by Popularity


*1 - Onym Class Priority Order*
We believe it should be possible to give different priority to different
class of nyms (hypernym/hyponym/synonym).
Specifically we do believe that should be possible to model this priority
in scoring:

*Original Query Term > True Synonym > Hyponym > Hypernym .*

Additional benefit could be gained if such inequality could be customised
based on user requirements.
*i.e.*
Adding different shades of nyms and slighly different ordering :
Original Query Term > True Synonym > Hyponym > 2 level hyponym > Hypernym .

*2 - Onyms within a Class Ranked by Popularity*

Within the same class we believe we need to favour the most popular
(highest Document Frequency) onyms.
i.e. within true synonyms we'll favour the most popular one.
The same within hyponyms or hypernyms.
Generally within an Onym class we want to rank higher the terms with higher
document frequency.

*Proposed Solution*
The proposed solution is to score the different onyms in this way :

*Original Query Term -> *IDFQueryTerm
*True Synonym (boost: 1.0)* ->  IDFQueryTerm * 1/(1+IDFSynonym)
*Hyponym (boost<1.0)*->  IDFQueryTerm * 1/(1+IDFHyponym)
*Hypernym (boost<1.0)* ->  IDFQueryTerm * 1/(1+IDFHypernym)

You may noticed the introduction of the boost factor.
This is the key point of the Onym classification.
All the onyms with the same boost will belong to the same class.
This gives the user the flexibility of ranking the different Onyms classes
based on their preference.
The boost solves the problem 1 (*Onym Class Priority Order*).
Multiplying the original term IDF with the second part of the formula fixes
problem 2 (*Onyms within a Class Ranked by Popularity*) and guarantee the
original term to win anyway.

*Implementation*
The suggested implementation will cover different areas :
- implement the scoring logic through blended DF/ proxy term stats/ proxy
similarity (it must be investigated the best path to implement the designed
scoring)
- Give the user a configuration file to model the Onyms. A first modality
is already available through [2]. A first improvement could be to implement
the support for taxonomies such as :
/big cats/lion-panthera leo/simba-kimba.
A final solution will allow an integration with custom knowledge bases,
wordnet, ect ect
- what about performance ? you could add a configuration parameters that
cut the query expansion based on a boost threshold. We can imagine the
boost as the distance from the original concept, so the user should be able
to cut down the expanded terms to favour performances.

[1] https://issues.apache.org/jira/browse/SOLR-11662,
https://github.com/elastic/elasticsearch/pull/35422

[2] https://issues.apache.org/jira/browse/SOLR-12238
--
Alessandro Benedetti
Search Consultant, R Software Engineer, Director
www.sease.io


On Wed, Nov 21, 2018 at 2:34 AM Michael Sokolov  wrote:

> This is a great idea. It would also be compelling to modify the term
> frequency using this deboosting so that stacked indexed terms can be
> weighted according to their closeness to the original term.
>
> On Tue, Nov 20, 2018, 2:19 PM jim ferenczi 
>> Sorry for the late reply,
>>
>> > So perhaps one way forward to contribute this sort of thing into Lucene
>> is we could implement additional QueryBuilder implementations that provide
>> such functionality?
>>
>> I am not sure, I mentioned Solr and ES because I thought it was about
>> adding taxonomies and complex expansion mechanisms to query builders but I
>> wonder if we can have a simple
>> mechanism to just (de)boost stacked tokens in the QueryBuilder. It could
>> be a new attribute that token filters would use when they produce stacked
>> tokens and that the QueryBuilder checks when he builds the SynonymQuery. We
>> already have a TermFrequencyAttribute to alter the 

[GitHub] lucene-solr issue #503: LUCENE-8571: Don't block on FrozenBufferedUpdates#ap...

[s1monw]

Github user s1monw commented on the issue:

https://github.com/apache/lucene-solr/pull/503
  
@mikemccand can you take a look


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #503: LUCENE-8571: Don't block on FrozenBufferedUpd...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/503#discussion_r235314171
  
--- Diff: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java ---
@@ -2607,7 +2607,9 @@ synchronized long 
publishFrozenUpdates(FrozenBufferedUpdates packet) {
 // Do this as an event so it applies higher in the stack when we are 
not holding DocumentsWriterFlushQueue.purgeLock:
 eventQueue.add(w -> {
   try {
-packet.apply(w);
+// we call tryApply here since we don't want to block if a refresh 
or a flush is already applying the
+// packet. The flush will retry this packet anyway to ensure all 
of the are applied
--- End diff --

s/all of the/all of them/


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 1981 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/1981/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.6/8/consoleText

[repro] Revision: bbe9511894340df14b938565cf3575540ca2f21b

[repro] Ant options: -DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9
[repro] Repro line:  ant test  -Dtestcase=MoveReplicaTest -Dtests.method=test 
-Dtests.seed=B71BA1F6E526F6DA -Dtests.multiplier=2 -Dtests.locale=es-UY 
-Dtests.timezone=Europe/Stockholm -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=IndexSizeTriggerTest 
-Dtests.method=testSplitIntegration -Dtests.seed=B71BA1F6E526F6DA 
-Dtests.multiplier=2 -Dtests.locale=et-EE -Dtests.timezone=SystemV/EST5 
-Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
08dd681f0febcf73af94b47ea742294bf4dd8701
[repro] git fetch
[repro] git checkout bbe9511894340df14b938565cf3575540ca2f21b

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   IndexSizeTriggerTest
[repro]   MoveReplicaTest
[repro] ant compile-test

[...truncated 3580 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 
-Dtests.class="*.IndexSizeTriggerTest|*.MoveReplicaTest" 
-Dtests.showOutput=onerror 
-DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9 
-Dtests.seed=B71BA1F6E526F6DA -Dtests.multiplier=2 -Dtests.locale=et-EE 
-Dtests.timezone=SystemV/EST5 -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 30192 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.MoveReplicaTest
[repro]   2/5 failed: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
[repro] git checkout 08dd681f0febcf73af94b47ea742294bf4dd8701

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Call for help: moving from ant build to gradle

[Diego Ceccarelli (BLOOMBERG/ LONDON)]

Hi all, 

I just noticed this thread I would be happy to help if I can. 

I checked out the jira/gradle branch but I get conflicts if I try to rebase on 
top of the current master.. 

Do you have a list of subtasks for porting solr to gradle? 

Thanks,
Diego  


From: dev@lucene.apache.org At: 11/05/18 22:34:50To:  dev@lucene.apache.org
Subject: Re: Call for help: moving from ant build to gradle

Edward:

Of course. You may have to coordinate how to get your contributions
added to the patch is all.

I'd coordinate with Dat first though just for efficiency's sake. Just
work with the branch/gradle version of the code from the main Git
repo.
On Mon, Nov 5, 2018 at 2:22 PM Edward Ribeiro  wrote:
>
> Is this work open to contribution of non committers?
>
> Edward
>
>
> Em seg, 5 de nov de 2018 15:01, Gus Heck >
>> I'm quite fond of gradle, and even wrote a very simple plugin for uploading 
and downloading solr configs to zookeeper from gradle. +1 to use gradle.
>>
>> I'll definitely check it out and give it a whirl, maybe I'll help some if I 
can.
>>
>> On Sun, Nov 4, 2018 at 2:13 PM Đạt Cao Mạnh  wrote:
>>>
>>> Hi guys,
>>>
>>> Recently, I had a chance of working on modifying different build.xml of our 
project. To be honest that was a painful experience, especially the number of 
steps for adding a new module in our project. We reach the limitation point of 
Ant and moving to Gradle seems a good option since it has been widely used in 
many projects. There are several benefits of the moving here that I would like 
to mention
>>> * The capability of caching result in Gradle make running task much faster. 
I.e: rerunning forbiddenApi check in Gradle only takes 5 seconds (comparing to 
more than a minute of Ant).
>>> * Adding modules is much easier now.
>>> * Adding dependencies is a pleasure now since we don't have to run ant 
clean-idea and ant idea all over again.
>>> * Natively supported by different IDEs.
>>>
>>> On my very boring long flight from Montreal back to Vietnam, I tried to 
convert the Lucene/Solr Ant to Gradle, I finally achieved something here by 
being able to import project and run tests natively from IntelliJ IDEA (branch 
jira/gradle).
>>>
>>> I'm converting ant precommit for Lucene to Gradle. But there are a lot of 
things need to be done here and my limitation understanding in our Ant build 
and Gradle may make the work take a lot of time to finish.
>>>
>>> Therefore, I really need help from the community to finish the work and we 
will be able to move to a totally new, modern, powerful build tool.
>>>
>>> Thanks!
>>>
>>
>>
>> --
>> http://www.the111shift.com

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.6-Windows (64bit/jdk-10.0.1) - Build # 3 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Windows/3/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseParallelGC

5 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.TriggerSetPropertiesIntegrationTest.testSetProperties

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([2634B4FA167B1AF8:4D5063B3A5568F7C]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertTrue(Assert.java:54)
at 
org.apache.solr.cloud.autoscaling.TriggerSetPropertiesIntegrationTest.testSetProperties(TriggerSetPropertiesIntegrationTest.java:111)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  
org.apache.solr.client.solrj.impl.CloudSolrClientTest.testVersionsAreReturned

Error Message:
Error from server at http://127.0.0.1:50033/solr/collection1_shard2_replica_n3: 
Expected mime type application/octet-stream but got text/html.   
 

[jira] [Commented] (LUCENE-8562) Speed up merging segments of points with data dimensions

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694575#comment-16694575
 ] 

Adrien Grand commented on LUCENE-8562:
--

I think we could just fix the assert. :) OK let's get this in as-is, I agree 
with you that it has the merit of compressing the same way as today. We can 
look later into simplifying this.

> Speed up merging segments of points with data dimensions
> 
>
> Key: LUCENE-8562
> URL: https://issues.apache.org/jira/browse/LUCENE-8562
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: master (8.0), 7.7
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8562.patch, LUCENE-8562.patch, LUCENE-8562.patch
>
>
> Currently when merging segments of points with data dimensions, all 
> dimensions are sorted and carried over down the tree even though only 
> indexing dimensions are needed to build the BKD tree. This is needed so leaf 
> node data can be compressed by common prefix.
> But when using _MutablePointValues_, this ordering is done at the leaf level 
> so we can se a similar approach from data dimensions and delay the sorting at 
> leaf level. This seems to speed up indexing time as well as reduce the 
> storage needed for building the index.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #503: LUCENE-8571: Don't block on FrozenBufferedUpd...

[s1monw]

GitHub user s1monw opened a pull request:

https://github.com/apache/lucene-solr/pull/503

LUCENE-8571: Don't block on FrozenBufferedUpdates#apply during 
IW#processEvents

While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to
ensure we apply all frozen update packages before we return.
Today we execute the apply method in a blocking fashion which is unnecessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary 
updates
were we can continue with other work instead of waiting.
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/s1monw/lucene-solr 
optimize_apply_frozen_updates

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/503.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #503


commit caeb68509a53766963bbcb1a96ef204ba6cb1492
Author: Simon Willnauer 
Date:   2018-11-21T09:22:41Z

LUCENE-8571: Don't block on FrozenBufferedUpdates#apply during 
IW#processEvents

While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to
ensure we apply all frozen update packages before we return.
Today we execute the apply method in a blocking fashion which is unncessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary 
updates
were we can continue with other work instead of waiting.
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.




---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13003) Query Result Cache does not honour maxRamBytes parameter

[Shalin Shekhar Mangar (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694354#comment-16694354
 ] 

Shalin Shekhar Mangar commented on SOLR-13003:
--

The default queryResultCache implementation is the LRUCache which supports 
maxRamMB and that is tested. You are using the FastLRUCache for query result 
cache. This combination is not tested. I think that's where the problem lies. 
Do you mind switching query result cache back to LRUCache. Was there are a 
reason you made the switch?

I'll look into how query result cache using FastLRUCache triggers this bug and 
either fix it or disable the specific combination.

> Query Result Cache does not honour maxRamBytes parameter
> 
>
> Key: SOLR-13003
> URL: https://issues.apache.org/jira/browse/SOLR-13003
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 7.3.1
>Reporter: Cetra Free
>Priority: Major
> Attachments: CLRU-logging.patch, lrucacheexpanded.png, 
> lrucachemaxmb.png, solr-core-7.3.1-SNAPSHOT.jar, solrconfig.xml
>
>
> When using the maxRamBytes parameter with the queryResultCache directive, we 
> have seen the retained size of the cache orders of magnitude larger than what 
> is configured.
> Please see attached VisualVM output which shows the retained size is about 
> 1.5gb, but the maxRamBytes is set to 64mb.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 1980 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/1980/

[...truncated 49 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-7.x/1033/consoleText

[repro] Revision: a4e95f39be9d7f107b9027eb950775dc59406932

[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=raceConditionOnDeleteAndRegisterReplica 
-Dtests.seed=BE834F3AB20EE7F4 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=da-DK -Dtests.timezone=Europe/London -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=DeleteReplicaTest 
-Dtests.method=deleteLiveReplicaTest -Dtests.seed=BE834F3AB20EE7F4 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=da-DK 
-Dtests.timezone=Europe/London -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] Repro line:  ant test  -Dtestcase=LeaderElectionIntegrationTest 
-Dtests.method=testSimpleSliceLeaderElection -Dtests.seed=BE834F3AB20EE7F4 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sr-Latn 
-Dtests.timezone=Asia/Gaza -Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
492c3440def2112fe76c99b691cc073ca6511f62
[repro] git fetch
[repro] git checkout a4e95f39be9d7f107b9027eb950775dc59406932

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   DeleteReplicaTest
[repro]   LeaderElectionIntegrationTest
[repro] ant compile-test

[...truncated 3582 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=10 
-Dtests.class="*.DeleteReplicaTest|*.LeaderElectionIntegrationTest" 
-Dtests.showOutput=onerror  -Dtests.seed=BE834F3AB20EE7F4 -Dtests.multiplier=2 
-Dtests.slow=true -Dtests.locale=da-DK -Dtests.timezone=Europe/London 
-Dtests.asserts=true -Dtests.file.encoding=US-ASCII

[...truncated 14545 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   0/5 failed: org.apache.solr.cloud.LeaderElectionIntegrationTest
[repro]   1/5 failed: org.apache.solr.cloud.DeleteReplicaTest
[repro] git checkout 492c3440def2112fe76c99b691cc073ca6511f62

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8562) Speed up merging segments of points with data dimensions

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694461#comment-16694461
 ] 

Adrien Grand commented on LUCENE-8562:
--

bq. I think for common prefix to work you need all dimensions to be computed 
but not 100% sure

Right, computing common prefixes is important. I was only considering removing 
the logic that computes the cardinality of the leading uncommon byte and 
potentially selects data dimensions as a "sortedDim" as this adds some tricky 
code.

> Speed up merging segments of points with data dimensions
> 
>
> Key: LUCENE-8562
> URL: https://issues.apache.org/jira/browse/LUCENE-8562
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: master (8.0), 7.7
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8562.patch, LUCENE-8562.patch
>
>
> Currently when merging segments of points with data dimensions, all 
> dimensions are sorted and carried over down the tree even though only 
> indexing dimensions are needed to build the BKD tree. This is needed so leaf 
> node data can be compressed by common prefix.
> But when using _MutablePointValues_, this ordering is done at the leaf level 
> so we can se a similar approach from data dimensions and delay the sorting at 
> leaf level. This seems to speed up indexing time as well as reduce the 
> storage needed for building the index.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #502: LUCENE-8569: Never count soft-deletes if read...

[jpountz]

Github user jpountz commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/502#discussion_r235312499
  
--- Diff: lucene/core/src/java/org/apache/lucene/index/IndexWriter.java ---
@@ -4398,42 +4429,41 @@ private int mergeMiddle(MergePolicy.OneMerge merge, 
MergePolicy mergePolicy) thr
 
   // Let the merge wrap readers
   List mergeReaders = new ArrayList<>();
-  int softDeleteCount = 0;
+  Counter softDeleteCount = Counter.newCounter(false);
   for (int r = 0; r < merge.readers.size(); r++) {
 SegmentReader reader = merge.readers.get(r);
 CodecReader wrappedReader = merge.wrapForMerge(reader);
 validateMergeReader(wrappedReader);
 if (softDeletesEnabled) {
+
   if (reader != wrappedReader) { // if we don't have a wrapped 
reader we won't preserve any soft-deletes
 Bits hardLiveDocs = merge.hardLiveDocs.get(r);
-Bits wrappedLiveDocs = wrappedReader.getLiveDocs();
-int hardDeleteCount = 0;
-DocIdSetIterator softDeletedDocs = 
DocValuesFieldExistsQuery.getDocValuesDocIdSetIterator(config.getSoftDeletesField(),
 wrappedReader);
-if (softDeletedDocs != null) {
-  int docId;
-  while ((docId = softDeletedDocs.nextDoc()) != 
DocIdSetIterator.NO_MORE_DOCS) {
-if (wrappedLiveDocs == null || wrappedLiveDocs.get(docId)) 
{
-  if (hardLiveDocs == null || hardLiveDocs.get(docId)) {
-softDeleteCount++;
-  } else {
-hardDeleteCount++;
+if (hardLiveDocs != null) { // we only need to do this 
accounting if we have mixed deletes
+  Bits wrappedLiveDocs = wrappedReader.getLiveDocs();
+  Counter hardDeleteCounter = Counter.newCounter(false);
+  countSoftDeletes(wrappedReader, wrappedLiveDocs, 
hardLiveDocs, softDeleteCount, hardDeleteCounter);
+  int hardDeleteCount = 
Math.toIntExact(hardDeleteCounter.get());
+  // Wrap the wrapped reader again if we have excluded some 
hard-deleted docs
+  if (hardLiveDocs != null && hardDeleteCount > 0) {
--- End diff --

hardLiveDocs is always non-null at this stage?


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8571) Don't block on FrozenBufferedUpdates#apply during IW#processEvents

[Simon Willnauer (JIRA)]

Simon Willnauer created LUCENE-8571:
---

 Summary: Don't block on FrozenBufferedUpdates#apply during 
IW#processEvents
 Key: LUCENE-8571
 URL: https://issues.apache.org/jira/browse/LUCENE-8571
 Project: Lucene - Core
  Issue Type: Improvement
Affects Versions: 7.5, master (8.0)
Reporter: Simon Willnauer


While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to 
ensure we apply all frozen update packages before we return. 
Today we execute the apply method in a blocking fashion which is unnecessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary updates
were we can continue with other work instead of waiting. 
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8562) Speed up merging segments of points with data dimensions

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694430#comment-16694430
 ] 

Lucene/Solr QA commented on LUCENE-8562:


| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red}  0m  
0s{color} | {color:red} The patch doesn't appear to include any new or modified 
tests. Please justify why no new tests are needed for this patch. Also please 
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  0m 
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  0m 28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  0m 28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  0m 28s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 13m 
39s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m  0s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-8562 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12948817/LUCENE-8562.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 492c344 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | 1.8.0_191 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-LUCENE-Build/123/testReport/ |
| modules | C: lucene/core U: lucene/core |
| Console output | 
https://builds.apache.org/job/PreCommit-LUCENE-Build/123/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Speed up merging segments of points with data dimensions
> 
>
> Key: LUCENE-8562
> URL: https://issues.apache.org/jira/browse/LUCENE-8562
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: master (8.0), 7.7
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8562.patch, LUCENE-8562.patch
>
>
> Currently when merging segments of points with data dimensions, all 
> dimensions are sorted and carried over down the tree even though only 
> indexing dimensions are needed to build the BKD tree. This is needed so leaf 
> node data can be compressed by common prefix.
> But when using _MutablePointValues_, this ordering is done at the leaf level 
> so we can se a similar approach from data dimensions and delay the sorting at 
> leaf level. This seems to speed up indexing time as well as reduce the 
> storage needed for building the index.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8216) Better cross-field scoring

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694447#comment-16694447
 ] 

ASF subversion and git services commented on LUCENE-8216:
-

Commit fd96bc5ca6b1cf0c24953fb7b35937e403846440 in lucene-solr's branch 
refs/heads/master from [~jim.ferenczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=fd96bc5 ]

LUCENE-8216: Added a new BM25FQuery in sandbox to blend statistics across 
several fields using the BM25F formula


> Better cross-field scoring
> --
>
> Key: LUCENE-8216
> URL: https://issues.apache.org/jira/browse/LUCENE-8216
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: LUCENE-8216.patch, LUCENE-8216.patch
>
>
> I'd like Lucene to have better support for scoring across multiple fields. 
> Today we have BlendedTermQuery which tries to help there but it probably 
> tries to do too much on some aspects (handling cross-field term queries AND 
> synonyms) and too little on other ones (it tries to merge index-level 
> statistics, but not per-document statistics like tf and norm).
> Maybe we could implement something like BM25F so that queries across multiple 
> fields would retain the benefits of BM25 like the fact that the impact of the 
> term frequency saturates quickly, which is not the case with BlendedTermQuery 
> if you have occurrences across many fields.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8562) Speed up merging segments of points with data dimensions

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694484#comment-16694484
 ] 

Ignacio Vera commented on LUCENE-8562:
--

I got you, I will play with the idea.

> Speed up merging segments of points with data dimensions
> 
>
> Key: LUCENE-8562
> URL: https://issues.apache.org/jira/browse/LUCENE-8562
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: master (8.0), 7.7
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8562.patch, LUCENE-8562.patch
>
>
> Currently when merging segments of points with data dimensions, all 
> dimensions are sorted and carried over down the tree even though only 
> indexing dimensions are needed to build the BKD tree. This is needed so leaf 
> node data can be compressed by common prefix.
> But when using _MutablePointValues_, this ordering is done at the leaf level 
> so we can se a similar approach from data dimensions and delay the sorting at 
> leaf level. This seems to speed up indexing time as well as reduce the 
> storage needed for building the index.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8216) Better cross-field scoring

[Jim Ferenczi (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694443#comment-16694443
 ] 

Jim Ferenczi commented on LUCENE-8216:
--

{quote}

Should we require that weights are greater than or equal to 1 so that ttf is 
guaranteed to be greater than or equal to df?

{quote}

Yes this is required because we keep the max of the df and the sum of the ttf. 
I'll add a check.

 

{quote}

In BM25Query.rewrite(), I think you need to put the 'single field, single term' 
case before the 'single field' case?

{quote}

 

Good catch, thanks.

 

 

> Better cross-field scoring
> --
>
> Key: LUCENE-8216
> URL: https://issues.apache.org/jira/browse/LUCENE-8216
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: LUCENE-8216.patch, LUCENE-8216.patch
>
>
> I'd like Lucene to have better support for scoring across multiple fields. 
> Today we have BlendedTermQuery which tries to help there but it probably 
> tries to do too much on some aspects (handling cross-field term queries AND 
> synonyms) and too little on other ones (it tries to merge index-level 
> statistics, but not per-document statistics like tf and norm).
> Maybe we could implement something like BM25F so that queries across multiple 
> fields would retain the benefits of BM25 like the fact that the impact of the 
> term frequency saturates quickly, which is not the case with BlendedTermQuery 
> if you have occurrences across many fields.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Jan Høydahl]

> bq. I don't want to see file limit warnings on a dev laptop
> 
> "If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to
> false in your profile or solr.in.sh"
> 
> Not having these limits set properly has caused weird errors for a lot
> of people, but I agree they are annoying at times so there's a way to
> shut them off.

Note my wording - "on a dev laptop". We can find a way to distinguish between 
dev/prod
usage and only warn about file descriptors, suspiciously low heap, lack of SSL 
etc if
Solr is bound to a public IP address.

> The only logging config files you should have to worry about are in
> ./solr/server/resources.
> 
> But yeah, there are several Solr JIRAs lying around that I'll break
> some time loose _sometime_ to work on  to try to make logging more
> rational. Unfortunately as Shawn says, there's always tension. When
> running "normally", I pretty much only want to see WARN messages. Then
> every time I want to diagnose something, I want to see _everything_.
> And telling the user to change their logging and call me again next
> time the problem happens is not very satisfying for anyone.

It is super easy to change the log level of one particular class or package
selectively, that's why we have log4j2.xml and also the Admin's Logging levels 
menu.
The occational need to see all of the 150 jar files loaded on startup should 
not be
solved by spewing out that for first-time users of Solr who really just care 
for whether
the server started OK or not.

Agree that adding commented-out snippets in the logging config could be a
good way to quickly let people debug "ClassNotFound issues", "Indexing issues",
"Faceting issues", "Security issues" etc. We could have pre-canned configs that
would upper the logging of classes of particular interest for each case.

Jan

[jira] [Commented] (LUCENE-8216) Better cross-field scoring

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8216?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694458#comment-16694458
 ] 

ASF subversion and git services commented on LUCENE-8216:
-

Commit 08dd681f0febcf73af94b47ea742294bf4dd8701 in lucene-solr's branch 
refs/heads/master from [~jim.ferenczi]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=08dd681 ]

LUCENE-8216: improve error message when field weight is invalid


> Better cross-field scoring
> --
>
> Key: LUCENE-8216
> URL: https://issues.apache.org/jira/browse/LUCENE-8216
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Major
> Fix For: master (8.0)
>
> Attachments: LUCENE-8216.patch, LUCENE-8216.patch
>
>
> I'd like Lucene to have better support for scoring across multiple fields. 
> Today we have BlendedTermQuery which tries to help there but it probably 
> tries to do too much on some aspects (handling cross-field term queries AND 
> synonyms) and too little on other ones (it tries to merge index-level 
> statistics, but not per-document statistics like tf and norm).
> Maybe we could implement something like BM25F so that queries across multiple 
> fields would retain the benefits of BM25 like the fact that the impact of the 
> term frequency saturates quickly, which is not the case with BlendedTermQuery 
> if you have occurrences across many fields.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #503: LUCENE-8571: Don't block on FrozenBufferedUpdates#ap...

[s1monw]

Github user s1monw commented on the issue:

https://github.com/apache/lucene-solr/pull/503
  
I recommend to look at the diff with `?w=1` -> 
https://github.com/apache/lucene-solr/pull/503/files?w=1


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8562) Speed up merging segments of points with data dimensions

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694530#comment-16694530
 ] 

Ignacio Vera commented on LUCENE-8562:
--

Removing that logic makes test unhappy. In particular if we have the same 
values in the index dimensions but different values in the data dimensions, 
then it will fail when writing block packed values::
{code:java}
int prefixLenSum = Arrays.stream(commonPrefixLengths).sum();
if (prefixLenSum == packedBytesLength)
{ 
   out.writeByte((byte) -1); }
else {
  assert commonPrefixLengths[sortedDim] < bytesPerDim;
...
}{code}
The assert complains as the commonPrefixLengths for the sorted dimension is 
equal to bytesPerDim.

> Speed up merging segments of points with data dimensions
> 
>
> Key: LUCENE-8562
> URL: https://issues.apache.org/jira/browse/LUCENE-8562
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/index
>Affects Versions: master (8.0), 7.7
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8562.patch, LUCENE-8562.patch
>
>
> Currently when merging segments of points with data dimensions, all 
> dimensions are sorted and carried over down the tree even though only 
> indexing dimensions are needed to build the BKD tree. This is needed so leaf 
> node data can be compressed by common prefix.
> But when using _MutablePointValues_, this ordering is done at the leaf level 
> so we can se a similar approach from data dimensions and delay the sorting at 
> leaf level. This seems to speed up indexing time as well as reduce the 
> storage needed for building the index.
>  
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.6-Linux (64bit/jdk-10.0.1) - Build # 10 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/10/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseG1GC

6 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:41797/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:41797/solr
at 
__randomizedtesting.SeedInfo.seed([DB75261A13246A07:1A855FB63E74A0A0]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:35205/solr

Stack Trace:
java.lang.AssertionError: 

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 383 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/383/

5 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.testTrigger

Error Message:


Stack Trace:
java.lang.AssertionError
at 
__randomizedtesting.SeedInfo.seed([5C9E384BEBCFF939:3F550EC972008A14]:0)
at org.junit.Assert.fail(Assert.java:92)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertFalse(Assert.java:68)
at org.junit.Assert.assertFalse(Assert.java:79)
at 
org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.testTrigger(NodeAddedTriggerTest.java:154)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.update.processor.TimeRoutedAliasUpdateProcessorTest.testPreemptiveCreation

Error Message:
Captured an uncaught exception in thread: Thread[id=913, 
name=TRA-preemptive-creation-633-thread-1, state=RUNNABLE, 
group=TGRP-TimeRoutedAliasUpdateProcessorTest]

Stack 

Re: SynonymQuery / Query Expansion Strategies Discussion

[Robert Muir]

I don't think we should put scoring stuff into the analysis chain like
this. It already has a laundry list of responsibilities.

Analysis chain can tell you the term is stacked or its a certain type
or occurs a certain number of times, but it shouldn't be supplying
things such as floating point boosts. That kind of scoring
manipulation needs to really happen in query parsing/somewhere else.

On 11/20/18, jim ferenczi  wrote:
> Sorry for the late reply,
>
>> So perhaps one way forward to contribute this sort of thing into Lucene
> is we could implement additional QueryBuilder implementations that provide
> such functionality?
>
> I am not sure, I mentioned Solr and ES because I thought it was about
> adding taxonomies and complex expansion mechanisms to query builders but I
> wonder if we can have a simple
> mechanism to just (de)boost stacked tokens in the QueryBuilder. It could be
> a new attribute that token filters would use when they produce stacked
> tokens and that the QueryBuilder checks when he builds the SynonymQuery. We
> already have a TermFrequencyAttribute to alter the frequency of a term when
> indexing so we could have the same mechanism for query term boosting ?
>
> Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
> dturnb...@opensourceconnections.com> a écrit :
>
>> Thanks Jim
>>
>> Yeah, now that I think about it - I agree that perhaps the simplest
>> option
>> would to create alternate query builders. I think there's a couple of
>> enhancement to the base class that would be nice, such as
>> - Some additional token attributes passed to newSynonymQuery, such as the
>> type (was this a synonym or hyponym or something else...)
>> - The ability to differentiate between the original query term and the
>> generated synonym terms
>> - Consistent support for phrases
>>
>> I think part of my goal too is to help people without the use of plugins.
>> As we often are in scenarios at OpenSource Connections where people won't
>> be able to use a plugin. In this case alternate expansions around
>> hypernyms/hyponyms/?... are a pretty frequent gap that search teams have
>> using Solr/Lucene/ES.
>>
>> So perhaps one way forward to contribute this sort of thing into Lucene
>> is
>> we could implement additional QueryBuilder implementations that provide
>> such functionality?
>>
>> Thanks
>> -Doug
>>
>> On Sat, Nov 17, 2018 at 3:41 PM jim ferenczi 
>> wrote:
>>
>>> You can easily customize the query that is used for synonyms in a custom
>>> QueryBuilder. The javadocs of the *newSynonymQuery* says "This is
>>> intended for subclasses that wish to customize the generated queries." so
>>> I
>>> don't think we need to do anything there. I agree that it is sometimes
>>> better to use something different than the SynonymQuery but in the
>>> general
>>> case it works as expected and can be combined with other terms
>>> naturally.
>>> The kind of customization you want to achieve could be done in a plugin
>>> (or
>>> in Solr or ES) that extends the QueryBuilder, you can also use custom
>>> token
>>> filters and alter the query the way you want. My point here is that the
>>> QueryBuilder should remain simple, you can add the complexity you want in
>>> a
>>> subclass.
>>> However I think there is another area we need to fix, the scoring of
>>> multi-terms synonyms is broken (compared to the SynonymQuery) and could
>>> be
>>> improved so we need something similar than the SynonymQuery that handles
>>> multi phrases.
>>>
>>>
>>> Le sam. 17 nov. 2018 à 07:19, Doug Turnbull <
>>> dturnb...@opensourceconnections.com> a écrit :
>>>
 Yes that is another good area (there are many). Although of course
 embeddings have their own challenges and complexities. (they often
 capture
 shared context, but not shared meaning).

 It's a data point though of something we'd want to include in such a
 framework, though not sure where it would go on the roadmap...

 On Sat, Nov 17, 2018 at 1:15 AM J. Delgado 
 wrote:

> What about the use of word embeddings (see
>
> https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa)
> to compute word similarity?
>
> On Sat, Nov 17, 2018 at 5:52 AM Doug Turnbull <
> dturnb...@opensourceconnections.com> wrote:
>
>> Hey folks,
>>
>> I wanted to open up a discussion about a change to the usage of
>> SynonymQuery. The goal here is to have a broader library of queries
>> that
>> can address other cases where related terms occupy the same position
>> but
>> don't have the same meaning (such as hypernyms, hyponyms, meronyms,
>> ambiguous terms, and other query expansion situations).
>>
>>
>> I bring this up because we've noticed (as I'm sure many of you have)
>> the pattern of clients jamming any related term into a synonyms file
>> and
>> being surprised with odd results. I like the idea of enforcing
>> "synonyms"
>> means 

[jira] [Resolved] (LUCENE-8571) Don't block on FrozenBufferedUpdates#apply during IW#processEvents

[Simon Willnauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer resolved LUCENE-8571.
-
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

> Don't block on FrozenBufferedUpdates#apply during IW#processEvents
> --
>
> Key: LUCENE-8571
> URL: https://issues.apache.org/jira/browse/LUCENE-8571
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: master (8.0), 7.7
>
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> While indexing we try to apply frozen deletes packages concurrently
> on indexing threads if necessary. This is done in an opaque way via
> IndexWriter#processEvents. Yet, when we commit or refresh we have to 
> ensure we apply all frozen update packages before we return. 
> Today we execute the apply method in a blocking fashion which is unnecessary
> when we are in a IndexWriter#processEvents context, we block indexing
> threads while they could just continue since it's already being applied.
> We also might wait in BufferedUpdatesStream when we apply all necessary 
> updates
> were we can continue with other work instead of waiting. 
> This change also tries to apply the packages that are not currently applied
> first in order to not unnecessarily block.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Call for help: moving from ant build to gradle

[Martin Gainty]

any reason for discriminating against a build system that supports customised 
lifecycles (e.g. maven)?

un saludo


From: Diego Ceccarelli (BLOOMBERG/ LONDON) 
Sent: Wednesday, November 21, 2018 5:30 AM
To: dev@lucene.apache.org
Subject: Re: Call for help: moving from ant build to gradle

Hi all,

I just noticed this thread I would be happy to help if I can.

I checked out the jira/gradle branch but I get conflicts if I try to rebase on 
top of the current master..

Do you have a list of subtasks for porting solr to gradle?

Thanks,
Diego




From: dev@lucene.apache.org At: 11/05/18 22:34:50
To: dev@lucene.apache.org
Subject: Re: Call for help: moving from ant build to gradle

Edward:

Of course. You may have to coordinate how to get your contributions
added to the patch is all.

I'd coordinate with Dat first though just for efficiency's sake. Just
work with the branch/gradle version of the code from the main Git
repo.
On Mon, Nov 5, 2018 at 2:22 PM Edward Ribeiro 
mailto:edward.ribe...@gmail.com>> wrote:
>
> Is this work open to contribution of non committers?
>
> Edward
>
>
> Em seg, 5 de nov de 2018 15:01, Gus Heck 
> mailto:gus.h...@gmail.com> escreveu:
>>
>> I'm quite fond of gradle, and even wrote a very simple plugin for uploading
and downloading solr configs to zookeeper from gradle. +1 to use gradle.
>>
>> I'll definitely check it out and give it a whirl, maybe I'll help some if I
can.
>>
>> On Sun, Nov 4, 2018 at 2:13 PM Đạt Cao Mạnh 
>> mailto:caomanhdat...@gmail.com>> wrote:
>>>
>>> Hi guys,
>>>
>>> Recently, I had a chance of working on modifying different build.xml of our
project. To be honest that was a painful experience, especially the number of
steps for adding a new module in our project. We reach the limitation point of
Ant and moving to Gradle seems a good option since it has been widely used in
many projects. There are several benefits of the moving here that I would like
to mention
>>> * The capability of caching result in Gradle make running task much faster.
I.e: rerunning forbiddenApi check in Gradle only takes 5 seconds (comparing to
more than a minute of Ant).
>>> * Adding modules is much easier now.
>>> * Adding dependencies is a pleasure now since we don't have to run ant
clean-idea and ant idea all over again.
>>> * Natively supported by different IDEs.
>>>
>>> On my very boring long flight from Montreal back to Vietnam, I tried to
convert the Lucene/Solr Ant to Gradle, I finally achieved something here by
being able to import project and run tests natively from IntelliJ IDEA (branch
jira/gradle).
>>>
>>> I'm converting ant precommit for Lucene to Gradle. But there are a lot of
things need to be done here and my limitation understanding in our Ant build
and Gradle may make the work take a lot of time to finish.
>>>
>>> Therefore, I really need help from the community to finish the work and we
will be able to move to a totally new, modern, powerful build tool.
>>>
>>> Thanks!
>>>
>>
>>
>> --
>> http://www.the111shift.com

-
To unsubscribe, e-mail: 
dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: 
dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

[Christophe Bismuth (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694827#comment-16694827
 ] 

Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:12 PM:
--

I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameter)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output:

{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2
no arcs in; skip pos=3
no arcs in; skip pos=4
no arcs in; skip pos=5
no arcs in; skip pos=6
  end: 1 nodes

  backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235
add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=7
{noformat}


was (Author: cbismuth):
I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameter)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below:

{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2
no 

[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

[Christophe Bismuth (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694827#comment-16694827
 ] 

Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:12 PM:
--

I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameter)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below:

{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2
no arcs in; skip pos=3
no arcs in; skip pos=4
no arcs in; skip pos=5
no arcs in; skip pos=6
  end: 1 nodes

  backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235
add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=7
{noformat}


was (Author: cbismuth):
I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameter)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below.
{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2

[JENKINS] Lucene-Solr-7.6-Linux (64bit/jdk-10.0.1) - Build # 11 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/11/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica

Error Message:
Timeout waiting for collection to become active Live Nodes: 
[127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:1_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr] Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/12)={   
"replicationFactor":"1",   "pullReplicas":"0",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"1",   "tlogReplicas":"0",   
"autoCreated":"true",   "policy":"c1",   "shards":{"shard1":{   
"replicas":{"core_node1":{   
"core":"testCreateCollectionAddReplica_shard1_replica_n1",   
"SEARCHER.searcher.maxDoc":0,   "SEARCHER.searcher.deletedDocs":0,  
 "INDEX.sizeInBytes":10240,   "node_name":"127.0.0.1:10004_solr",   
"state":"active",   "type":"NRT",   
"INDEX.sizeInGB":9.5367431640625E-6,   "SEARCHER.searcher.numDocs":0}}, 
  "range":"8000-7fff",   "state":"active"}}}

Stack Trace:
java.lang.AssertionError: Timeout waiting for collection to become active
Live Nodes: [127.0.0.1:10001_solr, 127.0.0.1:10004_solr, 127.0.0.1:1_solr, 
127.0.0.1:10002_solr, 127.0.0.1:10003_solr]
Last available state: 
DocCollection(testCreateCollectionAddReplica//clusterstate.json/12)={
  "replicationFactor":"1",
  "pullReplicas":"0",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"1",
  "tlogReplicas":"0",
  "autoCreated":"true",
  "policy":"c1",
  "shards":{"shard1":{
  "replicas":{"core_node1":{
  "core":"testCreateCollectionAddReplica_shard1_replica_n1",
  "SEARCHER.searcher.maxDoc":0,
  "SEARCHER.searcher.deletedDocs":0,
  "INDEX.sizeInBytes":10240,
  "node_name":"127.0.0.1:10004_solr",
  "state":"active",
  "type":"NRT",
  "INDEX.sizeInGB":9.5367431640625E-6,
  "SEARCHER.searcher.numDocs":0}},
  "range":"8000-7fff",
  "state":"active"}}}
at 
__randomizedtesting.SeedInfo.seed([29355AE7F0A65F11:A9153FC9E1E5B7B7]:0)
at 
org.apache.solr.cloud.CloudTestUtils.waitForState(CloudTestUtils.java:70)
at 
org.apache.solr.cloud.autoscaling.sim.TestSimPolicyCloud.testCreateCollectionAddReplica(TestSimPolicyCloud.java:123)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:110)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
 

[jira] [Commented] (SOLR-12775) Add a deprecated implementation of LowerCaseTokenizer

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694853#comment-16694853
 ] 

Lucene/Solr QA commented on SOLR-12775:
---

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
41s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  1m 31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  1m 31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  1m 31s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 53m 
55s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 59m 36s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-12775 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12948845/SOLR-12775.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 56cb42d |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | 1.8.0_191 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/231/testReport/ |
| modules | C: solr solr/core U: solr |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/231/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Add a deprecated implementation of LowerCaseTokenizer
> -
>
> Key: SOLR-12775
> URL: https://issues.apache.org/jira/browse/SOLR-12775
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-12775.patch, SOLR-12775.patch, SOLR-12775.patch, 
> SOLR-12775.patch
>
>
> LUCENE-8498 will remove LowerCaseTokenizer and LowerCaseTokenizerFactory from 
> lucene 8.  To make upgrading from Solr 7.x to Solr 8 easier for users who 
> have schemas that use LowerCaseTokenizerFactory, we should add a deprecated 
> copy of the code to the {{org.apache.solr.analysis}} package.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SynonymQuery / Query Expansion Strategies Discussion

[Michael Gibney]

On the analysis chain side, could the desired functionality be scoped to:
providing a framework (Attribute?) to express information about the
relationship between a derived token and its corresponding input? For
example, one might include information about:
1. corresponding input token (i.e., input token text?)
2. relationship between derived token and input (e.g., synonym, hyponym,
hypernym ... but perhaps not limited to these)
3. degree of confidence/weight in the derived token? This would represent a
concept distinct from "weight" for the purpose of scoring, and could thus
be appropriate to the analysis chain.
4. source/reason of token derivation relationship (e.g., specific ontology,
taxonomy, etc...)
5. 

This could provide all the information necessary to support custom indexing
strategies and/or query strategies, while remaining strictly focused on
analysis per se. This type of approach (if relationship info were recorded
in index, e.g. via Payload) could also support explicitly navigable facets
that are ontology-aware, or other potentially interesting things ...

Michael


On Wed, Nov 21, 2018 at 9:24 AM Doug Turnbull <
dturnb...@opensourceconnections.com> wrote:

> I agree there is a tension between analysis and query parser
> responsibilities (or external to how queries are constructed). I wonder
> what you'd think of making QueryBuilder more easily subclassible by passing
> more term metadata to newSynonymQuery (such as types etc). This would let
> you select an alt strategy (such as some of the scoring systems used in the
> query expansion paper https://arxiv.org/pdf/1708.00247.pdf). Or doing
> something with a term labeled a hyponym/hypernym in a QueryBuilder
> subclass..
>
> -Doug
>
> On Wed, Nov 21, 2018 at 8:09 AM Robert Muir  wrote:
>
>> I don't think we should put scoring stuff into the analysis chain like
>> this. It already has a laundry list of responsibilities.
>>
>> Analysis chain can tell you the term is stacked or its a certain type
>> or occurs a certain number of times, but it shouldn't be supplying
>> things such as floating point boosts. That kind of scoring
>> manipulation needs to really happen in query parsing/somewhere else.
>>
>> On 11/20/18, jim ferenczi  wrote:
>> > Sorry for the late reply,
>> >
>> >> So perhaps one way forward to contribute this sort of thing into Lucene
>> > is we could implement additional QueryBuilder implementations that
>> provide
>> > such functionality?
>> >
>> > I am not sure, I mentioned Solr and ES because I thought it was about
>> > adding taxonomies and complex expansion mechanisms to query builders
>> but I
>> > wonder if we can have a simple
>> > mechanism to just (de)boost stacked tokens in the QueryBuilder. It
>> could be
>> > a new attribute that token filters would use when they produce stacked
>> > tokens and that the QueryBuilder checks when he builds the
>> SynonymQuery. We
>> > already have a TermFrequencyAttribute to alter the frequency of a term
>> when
>> > indexing so we could have the same mechanism for query term boosting ?
>> >
>> > Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
>> > dturnb...@opensourceconnections.com> a écrit :
>> >
>> >> Thanks Jim
>> >>
>> >> Yeah, now that I think about it - I agree that perhaps the simplest
>> >> option
>> >> would to create alternate query builders. I think there's a couple of
>> >> enhancement to the base class that would be nice, such as
>> >> - Some additional token attributes passed to newSynonymQuery, such as
>> the
>> >> type (was this a synonym or hyponym or something else...)
>> >> - The ability to differentiate between the original query term and the
>> >> generated synonym terms
>> >> - Consistent support for phrases
>> >>
>> >> I think part of my goal too is to help people without the use of
>> plugins.
>> >> As we often are in scenarios at OpenSource Connections where people
>> won't
>> >> be able to use a plugin. In this case alternate expansions around
>> >> hypernyms/hyponyms/?... are a pretty frequent gap that search teams
>> have
>> >> using Solr/Lucene/ES.
>> >>
>> >> So perhaps one way forward to contribute this sort of thing into Lucene
>> >> is
>> >> we could implement additional QueryBuilder implementations that provide
>> >> such functionality?
>> >>
>> >> Thanks
>> >> -Doug
>> >>
>> >> On Sat, Nov 17, 2018 at 3:41 PM jim ferenczi 
>> >> wrote:
>> >>
>> >>> You can easily customize the query that is used for synonyms in a
>> custom
>> >>> QueryBuilder. The javadocs of the *newSynonymQuery* says "This is
>> >>> intended for subclasses that wish to customize the generated
>> queries." so
>> >>> I
>> >>> don't think we need to do anything there. I agree that it is sometimes
>> >>> better to use something different than the SynonymQuery but in the
>> >>> general
>> >>> case it works as expected and can be combined with other terms
>> >>> naturally.
>> >>> The kind of customization you want to achieve could be done in a
>> plugin
>> >>> (or
>> >>> in Solr or ES) 

Re: SynonymQuery / Query Expansion Strategies Discussion

[Doug Turnbull]

There's a lot of different topics here and ideas, so we captured the use
cases we see being discussed here as in this google doc
https://docs.google.com/document/d/1w4G9bEICJ1aarr3l7OodwR5aecPkbFTISOgymErpZfQ/edit#heading=h.pszpx5dpxq7a

Basically, we've seen 5 high-level use cases discussed
- Alt Labels (what SynonymQuery does well now)
- Synonyms (looser synonyms with close meaning that need to be scored
somehow - `notebook,laptop`)
- Taxonomies (hierarchies of concepts/terms `dress shoes\oxfords`)
- Ontologies / Knowledge Graphs (networks of concepts)
- Embeddings (distributed representations of a term)

It's a doc in progress, embeddings needs more work, and is probably the
hardest thing on the list. There's possible other

The goal isn't so much to make Lucene implement all of these (it would
create a lot of maintenance headaches to shove this all in), but some of it
is just defining practices / patterns / tools that enable these things in
Lucene-based search. Some may require no work, or some may require
supporting functionality.

-Doug

On Wed, Nov 21, 2018 at 9:23 AM Doug Turnbull <
dturnb...@opensourceconnections.com> wrote:

> I agree there is a tension between analysis and query parser
> responsibilities (or external to how queries are constructed). I wonder
> what you'd think of making QueryBuilder more easily subclassible by passing
> more term metadata to newSynonymQuery (such as types etc). This would let
> you select an alt strategy (such as some of the scoring systems used in the
> query expansion paper https://arxiv.org/pdf/1708.00247.pdf). Or doing
> something with a term labeled a hyponym/hypernym in a QueryBuilder
> subclass..
>
> -Doug
>
> On Wed, Nov 21, 2018 at 8:09 AM Robert Muir  wrote:
>
>> I don't think we should put scoring stuff into the analysis chain like
>> this. It already has a laundry list of responsibilities.
>>
>> Analysis chain can tell you the term is stacked or its a certain type
>> or occurs a certain number of times, but it shouldn't be supplying
>> things such as floating point boosts. That kind of scoring
>> manipulation needs to really happen in query parsing/somewhere else.
>>
>> On 11/20/18, jim ferenczi  wrote:
>> > Sorry for the late reply,
>> >
>> >> So perhaps one way forward to contribute this sort of thing into Lucene
>> > is we could implement additional QueryBuilder implementations that
>> provide
>> > such functionality?
>> >
>> > I am not sure, I mentioned Solr and ES because I thought it was about
>> > adding taxonomies and complex expansion mechanisms to query builders
>> but I
>> > wonder if we can have a simple
>> > mechanism to just (de)boost stacked tokens in the QueryBuilder. It
>> could be
>> > a new attribute that token filters would use when they produce stacked
>> > tokens and that the QueryBuilder checks when he builds the
>> SynonymQuery. We
>> > already have a TermFrequencyAttribute to alter the frequency of a term
>> when
>> > indexing so we could have the same mechanism for query term boosting ?
>> >
>> > Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
>> > dturnb...@opensourceconnections.com> a écrit :
>> >
>> >> Thanks Jim
>> >>
>> >> Yeah, now that I think about it - I agree that perhaps the simplest
>> >> option
>> >> would to create alternate query builders. I think there's a couple of
>> >> enhancement to the base class that would be nice, such as
>> >> - Some additional token attributes passed to newSynonymQuery, such as
>> the
>> >> type (was this a synonym or hyponym or something else...)
>> >> - The ability to differentiate between the original query term and the
>> >> generated synonym terms
>> >> - Consistent support for phrases
>> >>
>> >> I think part of my goal too is to help people without the use of
>> plugins.
>> >> As we often are in scenarios at OpenSource Connections where people
>> won't
>> >> be able to use a plugin. In this case alternate expansions around
>> >> hypernyms/hyponyms/?... are a pretty frequent gap that search teams
>> have
>> >> using Solr/Lucene/ES.
>> >>
>> >> So perhaps one way forward to contribute this sort of thing into Lucene
>> >> is
>> >> we could implement additional QueryBuilder implementations that provide
>> >> such functionality?
>> >>
>> >> Thanks
>> >> -Doug
>> >>
>> >> On Sat, Nov 17, 2018 at 3:41 PM jim ferenczi 
>> >> wrote:
>> >>
>> >>> You can easily customize the query that is used for synonyms in a
>> custom
>> >>> QueryBuilder. The javadocs of the *newSynonymQuery* says "This is
>> >>> intended for subclasses that wish to customize the generated
>> queries." so
>> >>> I
>> >>> don't think we need to do anything there. I agree that it is sometimes
>> >>> better to use something different than the SynonymQuery but in the
>> >>> general
>> >>> case it works as expected and can be combined with other terms
>> >>> naturally.
>> >>> The kind of customization you want to achieve could be done in a
>> plugin
>> >>> (or
>> >>> in Solr or ES) that extends the QueryBuilder, you can 

[jira] [Commented] (LUCENE-8571) Don't block on FrozenBufferedUpdates#apply during IW#processEvents

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694698#comment-16694698
 ] 

ASF subversion and git services commented on LUCENE-8571:
-

Commit 5ebfede368816f17b8478e7fb7927fc68773b42f in lucene-solr's branch 
refs/heads/branch_7x from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5ebfede ]

LUCENE-8571: Don't block on FrozenBufferedUpdates#apply during IW#processEvents

While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to
ensure we apply all frozen update packages before we return.
Today we execute the apply method in a blocking fashion which is unncessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary updates
were we can continue with other work instead of waiting.
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.


> Don't block on FrozenBufferedUpdates#apply during IW#processEvents
> --
>
> Key: LUCENE-8571
> URL: https://issues.apache.org/jira/browse/LUCENE-8571
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> While indexing we try to apply frozen deletes packages concurrently
> on indexing threads if necessary. This is done in an opaque way via
> IndexWriter#processEvents. Yet, when we commit or refresh we have to 
> ensure we apply all frozen update packages before we return. 
> Today we execute the apply method in a blocking fashion which is unnecessary
> when we are in a IndexWriter#processEvents context, we block indexing
> threads while they could just continue since it's already being applied.
> We also might wait in BufferedUpdatesStream when we apply all necessary 
> updates
> were we can continue with other work instead of waiting. 
> This change also tries to apply the packages that are not currently applied
> first in order to not unnecessarily block.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #494: ant compile error

[diegoceccarelli]

Github user diegoceccarelli commented on the issue:

https://github.com/apache/lucene-solr/pull/494
  
@cqvip-jq  this looks like a problem on your side. Try to remove the `ivy2` 
cache by removing the `.ivy2/cache` folder in your home and then run `ant 
ivy-bootstrap` and `ant dist` from the lucene-solr folder.. 


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #503: LUCENE-8571: Don't block on FrozenBufferedUpd...

[s1monw]

Github user s1monw closed the pull request at:

https://github.com/apache/lucene-solr/pull/503


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #503: LUCENE-8571: Don't block on FrozenBufferedUpdates#ap...

[s1monw]

Github user s1monw commented on the issue:

https://github.com/apache/lucene-solr/pull/503
  
pushed


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8569) Never count soft-deletes if reader has no hard-deletes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694719#comment-16694719
 ] 

ASF subversion and git services commented on LUCENE-8569:
-

Commit 8f76c1b0ca64cd65e05330de9379fed44cfe6a8f in lucene-solr's branch 
refs/heads/branch_7x from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=8f76c1b ]

LUCENE-8569: Never count soft-deletes if reader has no hard-deletes

Today we count the actual soft-deletes during a merge which is
unnecessary if there are no hard-deletes present. In this case, which
is considered to be the common case we can get accurate counts by substracting
the number of deleted docs in the wrapped reader from the number of soft-deletes
in that reader.


>  Never count soft-deletes if reader has no hard-deletes
> ---
>
> Key: LUCENE-8569
> URL: https://issues.apache.org/jira/browse/LUCENE-8569
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
>  Today we count the actual soft-deletes during a merge which is
> unnecessary if there are no hard-deletes present. In this case, which
> is considered to be the common case we can get accurate counts by 
> substracting
> the number of deleted docs in the wrapped reader from the number of 
> soft-deletes
> in that reader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

[Christophe Bismuth (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christophe Bismuth updated LUCENE-8548:
---
Attachment: testCyrillicWord.dot.png

> Reevaluate scripts boundary break in Nori's tokenizer
> -
>
> Key: LUCENE-8548
> URL: https://issues.apache.org/jira/browse/LUCENE-8548
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: testCyrillicWord.dot.png
>
>
> This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526:
> {noformat}
> Tokens are split on different character POS types (which seem to not quite 
> line up with Unicode character blocks), which leads to weird results for 
> non-CJK tokens:
> εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other 
> symbol) + μί/SL(Foreign language)
> ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + 
> ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol)
> Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + 
> лтичко/SL(Foreign language) + ̄/SY(Other symbol)
> don't is tokenized as don + t; same for don't (with a curly apostrophe).
> אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol)
> Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow
> While it is still possible to find these words using Nori, there are many 
> more chances for false positives when the tokens are split up like this. In 
> particular, individual numbers and combining diacritics are indexed 
> separately (e.g., in the Cyrillic example above), which can lead to a 
> performance hit on large corpora like Wiktionary or Wikipedia.
> Work around: use a character filter to get rid of combining diacritics before 
> Nori processes the text. This doesn't solve the Greek, Hebrew, or English 
> cases, though.
> Suggested fix: Characters in related Unicode blocks—like "Greek" and "Greek 
> Extended", or "Latin" and "IPA Extensions"—should not trigger token splits. 
> Combining diacritics should not trigger token splits. Non-CJK text should be 
> tokenized on spaces and punctuation, not by character type shifts. 
> Apostrophe-like characters should not trigger token splits (though I could 
> see someone disagreeing on this one).{noformat}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

[Christophe Bismuth (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694827#comment-16694827
 ] 

Christophe Bismuth commented on LUCENE-8548:


I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameters)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below.
{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2
no arcs in; skip pos=3
no arcs in; skip pos=4
no arcs in; skip pos=5
no arcs in; skip pos=6
  end: 1 nodes

  backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235
add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=7
{noformat}

> Reevaluate scripts boundary break in Nori's tokenizer
> -
>
> Key: LUCENE-8548
> URL: https://issues.apache.org/jira/browse/LUCENE-8548
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: testCyrillicWord.dot.png
>
>
> This was first reported in https://issues.apache.org/jira/browse/LUCENE-8526:
> {noformat}
> Tokens are split on different character POS types (which seem to not quite 
> line up with Unicode character blocks), which leads to weird results for 
> non-CJK tokens:
> εἰμί is tokenized as three tokens: ε/SL(Foreign language) + ἰ/SY(Other 
> symbol) + μί/SL(Foreign language)
> ka̠k̚t͡ɕ͈a̠k̚ is tokenized as ka/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol) + t/SL(Foreign language) + 
> ͡ɕ͈/SY(Other symbol) + a/SL(Foreign language) + ̠/SY(Other symbol) + 
> k/SL(Foreign language) + ̚/SY(Other symbol)
> Ба̀лтичко̄ is tokenized as ба/SL(Foreign language) + ̀/SY(Other symbol) + 
> лтичко/SL(Foreign language) + ̄/SY(Other symbol)
> don't is tokenized as don + t; same for don't (with a curly apostrophe).
> אוֹג׳וּ is tokenized as אוֹג/SY(Other symbol) + וּ/SY(Other symbol)
> Мoscow (with a Cyrillic М and the rest in Latin) is tokenized as м + oscow
> While it is still possible to find these words using Nori, there are many 
> more chances for false positives when the tokens are split up like this. In 
> particular, individual numbers and combining diacritics are indexed 
> separately (e.g., in the Cyrillic example above), which can lead to a 
> performance hit on large corpora like Wiktionary or Wikipedia.
> Work around: use a character filter to get rid of combining diacritics before 
> Nori processes the text. This doesn't solve the Greek, Hebrew, or English 
> cases, though.
> Suggested fix: Characters in related Unicode blocks—like "Greek" and 

[jira] [Comment Edited] (LUCENE-8548) Reevaluate scripts boundary break in Nori's tokenizer

[Christophe Bismuth (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694827#comment-16694827
 ] 

Christophe Bismuth edited comment on LUCENE-8548 at 11/21/18 3:11 PM:
--

I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameter)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below.
{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2
no arcs in; skip pos=3
no arcs in; skip pos=4
no arcs in; skip pos=5
no arcs in; skip pos=6
  end: 1 nodes

  backtrace: endPos=7 pos=7; 6 characters; last=1 cost=-235
add token=DictionaryToken("w" pos=6 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=5 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("c" pos=4 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=3 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("s" pos=2 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
add token=DictionaryToken("o" pos=1 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=7
{noformat}


was (Author: cbismuth):
I'm hacking around in the {{KoreanTokenizer}} class, but I'll need some 
mentoring to keep going on.

Here is what I've done so far:
 * Implement a Cyrillic test failure (see previous comment)
 * Locate the {{KoreanAnalyzer}} and {{KoreanTokenizer}} classes
 * Locate the {{ICUTokenizer}} and its {{CompositeBreakIterator}} class 
attribute (following UAX #29: Unicode Text Segmentation)
 * Try to make Ant {{nori}} module depend on {{icu}} module to try to reuse 
some {{ICUTokenizer}} logic parts (but I failed to tweak Ant scripts)
 * Enable verbose output (see output below)
 * Enable Graphiz ouput (see attached picture)
 * Debug step by step the 
{{org.apache.lucene.analysis.ko.KoreanTokenizer#parse}} method
 * Add a breakpoint in the {{DictionaryToken}} constructor to try to understand 
how and when tokens are built (I also played with {{outputUnknownUnigrams}} 
parameters)

I would need some code or documentation pointers when you have time.

!testCyrillicWord.dot.png!

Tokenizer verbose output below.
{noformat}
PARSE

  extend @ pos=0 char=м hex=43c
1 arcs in
UNKNOWN word len=1 1 wordIDs
  fromIDX=0: cost=138 (prevCost=0 wordCost=795 bgCost=138 spacePenalty=0) 
leftID=1793 leftPOS=SL)
**
  + cost=933 wordID=36 leftID=1793 leastIDX=0 toPos=1 toPos.idx=0

  backtrace: endPos=1 pos=1; 1 characters; last=0 cost=933
add token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)
  freeBefore pos=1
TEST-TestKoreanAnalyzer.testCyrillicWord-seed#[9AA9487A32EFEB]:incToken: 
return token=DictionaryToken("м" pos=0 length=1 posLen=1 type=UNKNOWN wordId=0 
leftID=1798)

PARSE

  extend @ pos=1 char=o hex=6f
1 arcs in
UNKNOWN word len=6 1 wordIDs
  fromIDX=0: cost=-1030 (prevCost=0 wordCost=795 bgCost=-1030 
spacePenalty=0) leftID=1793 leftPOS=SL)
**
  + cost=-235 wordID=30 leftID=1793 leastIDX=0 toPos=7 toPos.idx=0
no arcs in; skip pos=2

[jira] [Commented] (SOLR-9856) Collect metrics for shard replication and tlog replay on replicas

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694640#comment-16694640
 ] 

Christine Poerschke commented on SOLR-9856:
---

Hello [~ab] – when reading the 'Core (SolrCore) Registry' 
[section|https://lucene.apache.org/solr/guide/7_5/metrics-reporting.html#core-solrcore-registry]
 in the latest (7.5) Solr Reference Guide I noticed a "(TBD, SOLR-9856)" 
mention. Could I/you/someone go and remove it since this ticket here is 
complete, or should it stay and be replaced with a pointer to another open JIRA 
issue?

> Collect metrics for shard replication and tlog replay on replicas
> -
>
> Key: SOLR-9856
> URL: https://issues.apache.org/jira/browse/SOLR-9856
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.0
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: 6.4, 7.0
>
> Attachments: SOLR-9856.patch
>
>
> Using API from SOLR-4735 add metrics for tracking outgoing replication from 
> leader to shard replicas, and for tracking transaction log processing on 
> replicas.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-10.0.1) - Build # 3126 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3126/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseG1GC

7 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:41157/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:41157/solr
at 
__randomizedtesting.SeedInfo.seed([9BD74C28C5C8C9E9:5A273584E898034E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:41537/solr

Stack Trace:

[jira] [Commented] (LUCENE-8569) Never count soft-deletes if reader has no hard-deletes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694718#comment-16694718
 ] 

ASF subversion and git services commented on LUCENE-8569:
-

Commit 56cb42d200dd999342c4cd5c92fbfa271ba91153 in lucene-solr's branch 
refs/heads/master from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=56cb42d ]

LUCENE-8569: Never count soft-deletes if reader has no hard-deletes

Today we count the actual soft-deletes during a merge which is
unnecessary if there are no hard-deletes present. In this case, which
is considered to be the common case we can get accurate counts by substracting
the number of deleted docs in the wrapped reader from the number of soft-deletes
in that reader.


>  Never count soft-deletes if reader has no hard-deletes
> ---
>
> Key: LUCENE-8569
> URL: https://issues.apache.org/jira/browse/LUCENE-8569
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
>  Today we count the actual soft-deletes during a merge which is
> unnecessary if there are no hard-deletes present. In this case, which
> is considered to be the common case we can get accurate counts by 
> substracting
> the number of deleted docs in the wrapped reader from the number of 
> soft-deletes
> in that reader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-9.0.4) - Build # 23244 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23244/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:3/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:3/solr
at 
__randomizedtesting.SeedInfo.seed([45C46B08F8C0E291:843412A4D5902836]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:37153/solr

Stack Trace:

Re: SynonymQuery / Query Expansion Strategies Discussion

[Doug Turnbull]

Great thoughts Jim - +1 to your idea

One brainstorm I had, is taxonomies have a kind of 'ideal scoring' that I
think would lead to a different blending strategy for taxonomies than
synonyms.

If you have a taxonomy:

\shoes\dress_shoes\oxfords
\shoes\dress_shoes\wingtips
\shoes\lazy_shoes\loafers
\shoes\lazy_shoes\sketchers

This taxonomy states - if a document mentions 'oxfords', it's also
discussing the concept of dress shoes. If it only mentions 'wingtips' it
also is discussing dress shoes.

Thus ideally, the true document frequency of the parent concept 'dress
shoes' is the combination of the children. This is the number of documents
that discuss this concept.

You can repeat this for grandparent concepts. The number of documents with
'shoes' really is all the documents mentioning oxfords, wingtips, loafers,
sketchers, and the like...

We have implemented this idea at index time, with index-time semantic
expansion to inject the parent concepts. (manually put dress_shoes into
documents that just mention wingtips). This is mentioned in this blog post
https://opensourceconnections.com/blog/2016/12/23/elasticsearch-synonyms-patterns-taxonomies/
and
conference talk https://www.youtube.com/watch?v=90F30PS-884 This is
annoying and requires reindexing. Though it's the most accurate.

BUT I think a blended query-time query would capture the same semantics.
You basically want to score a taxonomy like the following. Imagine a user
query of wingtips, you could imagine 3 should clauses that blend at
different levels

- Search for the term 'wingtips' (lowest doc freq, smallest set)
- Search for parent & sibling concepts (the set of all dress shoes)
- Search for grandparent, aunt, uncle, cousins... (the set of all shoes,
highest df)

text:wingtips OR Blended(text:wingtips, text:oxfords, text:dress_shoes) OR
Blended(text:wingtips, text:oxfords, text:dress_shoes, text:sketchers,
text:loafers, ...)

Right now this can be accomplished by just issuing 3 SHOULD queries with 3
different query-time analyzers each with different synonym expansions
(exact user term, child => parent/sibling, child => parent, grandparent,
etc...). And maybe it should stay that way.

But this is why I think it's a 'yes AND', yes I think it would be a great
addition to have synonym weighting. AND I think there are blending
strategies that are specific to the use case.

-Doug



On Tue, Nov 20, 2018 at 9:34 PM Michael Sokolov  wrote:

> This is a great idea. It would also be compelling to modify the term
> frequency using this deboosting so that stacked indexed terms can be
> weighted according to their closeness to the original term.
>
> On Tue, Nov 20, 2018, 2:19 PM jim ferenczi 
Sorry for the late reply,
>>
>> > So perhaps one way forward to contribute this sort of thing into Lucene
>> is we could implement additional QueryBuilder implementations that provide
>> such functionality?
>>
>> I am not sure, I mentioned Solr and ES because I thought it was about
>> adding taxonomies and complex expansion mechanisms to query builders but I
>> wonder if we can have a simple
>> mechanism to just (de)boost stacked tokens in the QueryBuilder. It could
>> be a new attribute that token filters would use when they produce stacked
>> tokens and that the QueryBuilder checks when he builds the SynonymQuery. We
>> already have a TermFrequencyAttribute to alter the frequency of a term when
>> indexing so we could have the same mechanism for query term boosting ?
>>
>> Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
>> dturnb...@opensourceconnections.com> a écrit :
>>
> Thanks Jim
>>>
>>> Yeah, now that I think about it - I agree that perhaps the simplest
>>> option would to create alternate query builders. I think there's a couple
>>> of enhancement to the base class that would be nice, such as
>>> - Some additional token attributes passed to newSynonymQuery, such as
>>> the type (was this a synonym or hyponym or something else...)
>>> - The ability to differentiate between the original query term and the
>>> generated synonym terms
>>> - Consistent support for phrases
>>>
>>> I think part of my goal too is to help people without the use of
>>> plugins. As we often are in scenarios at OpenSource Connections where
>>> people won't be able to use a plugin. In this case alternate expansions
>>> around hypernyms/hyponyms/?... are a pretty frequent gap that search teams
>>> have using Solr/Lucene/ES.
>>>
>>> So perhaps one way forward to contribute this sort of thing into Lucene
>>> is we could implement additional QueryBuilder implementations that provide
>>> such functionality?
>>>
>>> Thanks
>>> -Doug
>>>
>>
>>> On Sat, Nov 17, 2018 at 3:41 PM jim ferenczi 
>>> wrote:
>>>
 You can easily customize the query that is used for synonyms in a
 custom QueryBuilder. The javadocs of the *newSynonymQuery* says "This
 is intended for subclasses that wish to customize the generated queries."
 so I don't think we need to do anything there. I agree that it is 

Re: SynonymQuery / Query Expansion Strategies Discussion

[Doug Turnbull]

Alessandro reading your post, I realized I made a mistake in that you'd
need to go both up and down the hierarchy when blending. When a user
searches for dress shoes, going down a level (or two) is just as important.
If a user searches for 'dress shoes' you also need hyponym terms.

This works out if you do an index time expansion (child terms get parent
terms injected) but doesn't work out if you want a 100% query time blending.

In this case, I think I would revise my blending idea to

- Search for the term 'wingtips' (lowest doc freq, smallest set)
- Search for the term 'wingtips' blended with all child terms
- Search for parent & sibling concepts (the set of all dress shoes)
- Search for grandparent, aunt, uncle, cousins... (the set of all shoes,
highest df)

In this case, I don't *think* need any special weighting, as the true doc
freq of each concept recreates the priority ordering you guys came up with.
That's pretty neat!

-Doug

On Wed, Nov 21, 2018 at 7:20 AM Doug Turnbull <
dturnb...@opensourceconnections.com> wrote:

> Great thoughts Jim - +1 to your idea
>
> One brainstorm I had, is taxonomies have a kind of 'ideal scoring' that I
> think would lead to a different blending strategy for taxonomies than
> synonyms.
>
> If you have a taxonomy:
>
> \shoes\dress_shoes\oxfords
> \shoes\dress_shoes\wingtips
> \shoes\lazy_shoes\loafers
> \shoes\lazy_shoes\sketchers
>
> This taxonomy states - if a document mentions 'oxfords', it's also
> discussing the concept of dress shoes. If it only mentions 'wingtips' it
> also is discussing dress shoes.
>
> Thus ideally, the true document frequency of the parent concept 'dress
> shoes' is the combination of the children. This is the number of documents
> that discuss this concept.
>
> You can repeat this for grandparent concepts. The number of documents with
> 'shoes' really is all the documents mentioning oxfords, wingtips, loafers,
> sketchers, and the like...
>
> We have implemented this idea at index time, with index-time semantic
> expansion to inject the parent concepts. (manually put dress_shoes into
> documents that just mention wingtips). This is mentioned in this blog post
> https://opensourceconnections.com/blog/2016/12/23/elasticsearch-synonyms-patterns-taxonomies/
>  and
> conference talk https://www.youtube.com/watch?v=90F30PS-884 This is
> annoying and requires reindexing. Though it's the most accurate.
>
> BUT I think a blended query-time query would capture the same semantics.
> You basically want to score a taxonomy like the following. Imagine a user
> query of wingtips, you could imagine 3 should clauses that blend at
> different levels
>
> - Search for the term 'wingtips' (lowest doc freq, smallest set)
> - Search for parent & sibling concepts (the set of all dress shoes)
> - Search for grandparent, aunt, uncle, cousins... (the set of all shoes,
> highest df)
>
> text:wingtips OR Blended(text:wingtips, text:oxfords, text:dress_shoes) OR
> Blended(text:wingtips, text:oxfords, text:dress_shoes, text:sketchers,
> text:loafers, ...)
>
> Right now this can be accomplished by just issuing 3 SHOULD queries with 3
> different query-time analyzers each with different synonym expansions
> (exact user term, child => parent/sibling, child => parent, grandparent,
> etc...). And maybe it should stay that way.
>
> But this is why I think it's a 'yes AND', yes I think it would be a great
> addition to have synonym weighting. AND I think there are blending
> strategies that are specific to the use case.
>
> -Doug
>
>
>
> On Tue, Nov 20, 2018 at 9:34 PM Michael Sokolov 
> wrote:
>
>> This is a great idea. It would also be compelling to modify the term
>> frequency using this deboosting so that stacked indexed terms can be
>> weighted according to their closeness to the original term.
>>
>> On Tue, Nov 20, 2018, 2:19 PM jim ferenczi >
> Sorry for the late reply,
>>>
>>> > So perhaps one way forward to contribute this sort of thing into
>>> Lucene is we could implement additional QueryBuilder implementations that
>>> provide such functionality?
>>>
>>> I am not sure, I mentioned Solr and ES because I thought it was about
>>> adding taxonomies and complex expansion mechanisms to query builders but I
>>> wonder if we can have a simple
>>> mechanism to just (de)boost stacked tokens in the QueryBuilder. It could
>>> be a new attribute that token filters would use when they produce stacked
>>> tokens and that the QueryBuilder checks when he builds the SynonymQuery. We
>>> already have a TermFrequencyAttribute to alter the frequency of a term when
>>> indexing so we could have the same mechanism for query term boosting ?
>>>
>>> Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
>>> dturnb...@opensourceconnections.com> a écrit :
>>>
>> Thanks Jim

 Yeah, now that I think about it - I agree that perhaps the simplest
 option would to create alternate query builders. I think there's a couple
 of enhancement to the base class that would be nice, such as
 - 

[jira] [Commented] (LUCENE-8571) Don't block on FrozenBufferedUpdates#apply during IW#processEvents

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694695#comment-16694695
 ] 

ASF subversion and git services commented on LUCENE-8571:
-

Commit 5f8855ee0bf57c875df8c108892e8d78 in lucene-solr's branch 
refs/heads/master from [~simonw]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=5f8855e ]

LUCENE-8571: Don't block on FrozenBufferedUpdates#apply during IW#processEvents

While indexing we try to apply frozen deletes packages concurrently
on indexing threads if necessary. This is done in an opaque way via
IndexWriter#processEvents. Yet, when we commit or refresh we have to
ensure we apply all frozen update packages before we return.
Today we execute the apply method in a blocking fashion which is unncessary
when we are in a IndexWriter#processEvents context, we block indexing
threads while they could just continue since it's already being applied.
We also might wait in BufferedUpdatesStream when we apply all necessary updates
were we can continue with other work instead of waiting.
This change also tries to apply the packages that are not currently applied
first in order to not unnecessarily block.


> Don't block on FrozenBufferedUpdates#apply during IW#processEvents
> --
>
> Key: LUCENE-8571
> URL: https://issues.apache.org/jira/browse/LUCENE-8571
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> While indexing we try to apply frozen deletes packages concurrently
> on indexing threads if necessary. This is done in an opaque way via
> IndexWriter#processEvents. Yet, when we commit or refresh we have to 
> ensure we apply all frozen update packages before we return. 
> Today we execute the apply method in a blocking fashion which is unnecessary
> when we are in a IndexWriter#processEvents context, we block indexing
> threads while they could just continue since it's already being applied.
> We also might wait in BufferedUpdatesStream when we apply all necessary 
> updates
> were we can continue with other work instead of waiting. 
> This change also tries to apply the packages that are not currently applied
> first in order to not unnecessarily block.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #502: LUCENE-8569: Never count soft-deletes if read...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/lucene-solr/pull/502


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8569) Never count soft-deletes if reader has no hard-deletes

[Simon Willnauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer resolved LUCENE-8569.
-
   Resolution: Fixed
Fix Version/s: 7.7
   master (8.0)

>  Never count soft-deletes if reader has no hard-deletes
> ---
>
> Key: LUCENE-8569
> URL: https://issues.apache.org/jira/browse/LUCENE-8569
> Project: Lucene - Core
>  Issue Type: Improvement
>Affects Versions: 7.5, master (8.0)
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: master (8.0), 7.7
>
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
>  Today we count the actual soft-deletes during a merge which is
> unnecessary if there are no hard-deletes present. In this case, which
> is considered to be the common case we can get accurate counts by 
> substracting
> the number of deleted docs in the wrapped reader from the number of 
> soft-deletes
> in that reader.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SynonymQuery / Query Expansion Strategies Discussion

[Doug Turnbull]

I agree there is a tension between analysis and query parser
responsibilities (or external to how queries are constructed). I wonder
what you'd think of making QueryBuilder more easily subclassible by passing
more term metadata to newSynonymQuery (such as types etc). This would let
you select an alt strategy (such as some of the scoring systems used in the
query expansion paper https://arxiv.org/pdf/1708.00247.pdf). Or doing
something with a term labeled a hyponym/hypernym in a QueryBuilder
subclass..

-Doug

On Wed, Nov 21, 2018 at 8:09 AM Robert Muir  wrote:

> I don't think we should put scoring stuff into the analysis chain like
> this. It already has a laundry list of responsibilities.
>
> Analysis chain can tell you the term is stacked or its a certain type
> or occurs a certain number of times, but it shouldn't be supplying
> things such as floating point boosts. That kind of scoring
> manipulation needs to really happen in query parsing/somewhere else.
>
> On 11/20/18, jim ferenczi  wrote:
> > Sorry for the late reply,
> >
> >> So perhaps one way forward to contribute this sort of thing into Lucene
> > is we could implement additional QueryBuilder implementations that
> provide
> > such functionality?
> >
> > I am not sure, I mentioned Solr and ES because I thought it was about
> > adding taxonomies and complex expansion mechanisms to query builders but
> I
> > wonder if we can have a simple
> > mechanism to just (de)boost stacked tokens in the QueryBuilder. It could
> be
> > a new attribute that token filters would use when they produce stacked
> > tokens and that the QueryBuilder checks when he builds the SynonymQuery.
> We
> > already have a TermFrequencyAttribute to alter the frequency of a term
> when
> > indexing so we could have the same mechanism for query term boosting ?
> >
> > Le dim. 18 nov. 2018 à 02:24, Doug Turnbull <
> > dturnb...@opensourceconnections.com> a écrit :
> >
> >> Thanks Jim
> >>
> >> Yeah, now that I think about it - I agree that perhaps the simplest
> >> option
> >> would to create alternate query builders. I think there's a couple of
> >> enhancement to the base class that would be nice, such as
> >> - Some additional token attributes passed to newSynonymQuery, such as
> the
> >> type (was this a synonym or hyponym or something else...)
> >> - The ability to differentiate between the original query term and the
> >> generated synonym terms
> >> - Consistent support for phrases
> >>
> >> I think part of my goal too is to help people without the use of
> plugins.
> >> As we often are in scenarios at OpenSource Connections where people
> won't
> >> be able to use a plugin. In this case alternate expansions around
> >> hypernyms/hyponyms/?... are a pretty frequent gap that search teams have
> >> using Solr/Lucene/ES.
> >>
> >> So perhaps one way forward to contribute this sort of thing into Lucene
> >> is
> >> we could implement additional QueryBuilder implementations that provide
> >> such functionality?
> >>
> >> Thanks
> >> -Doug
> >>
> >> On Sat, Nov 17, 2018 at 3:41 PM jim ferenczi 
> >> wrote:
> >>
> >>> You can easily customize the query that is used for synonyms in a
> custom
> >>> QueryBuilder. The javadocs of the *newSynonymQuery* says "This is
> >>> intended for subclasses that wish to customize the generated queries."
> so
> >>> I
> >>> don't think we need to do anything there. I agree that it is sometimes
> >>> better to use something different than the SynonymQuery but in the
> >>> general
> >>> case it works as expected and can be combined with other terms
> >>> naturally.
> >>> The kind of customization you want to achieve could be done in a plugin
> >>> (or
> >>> in Solr or ES) that extends the QueryBuilder, you can also use custom
> >>> token
> >>> filters and alter the query the way you want. My point here is that the
> >>> QueryBuilder should remain simple, you can add the complexity you want
> in
> >>> a
> >>> subclass.
> >>> However I think there is another area we need to fix, the scoring of
> >>> multi-terms synonyms is broken (compared to the SynonymQuery) and could
> >>> be
> >>> improved so we need something similar than the SynonymQuery that
> handles
> >>> multi phrases.
> >>>
> >>>
> >>> Le sam. 17 nov. 2018 à 07:19, Doug Turnbull <
> >>> dturnb...@opensourceconnections.com> a écrit :
> >>>
>  Yes that is another good area (there are many). Although of course
>  embeddings have their own challenges and complexities. (they often
>  capture
>  shared context, but not shared meaning).
> 
>  It's a data point though of something we'd want to include in such a
>  framework, though not sure where it would go on the roadmap...
> 
>  On Sat, Nov 17, 2018 at 1:15 AM J. Delgado  >
>  wrote:
> 
> > What about the use of word embeddings (see
> >
> >
> https://towardsdatascience.com/introduction-to-word-embedding-and-word2vec-652d0c2060fa
> )
> > to compute word similarity?
> >
> > On 

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4939 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4939/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

5 tests failed.
FAILED:  
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test

Error Message:
Node 127.0.0.1:65140_solr has 3 replicas. Expected num replicas : 2. state:  
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/11)={
   "pullReplicas":0,   "replicationFactor":1,   "shards":{ "shard2":{   
"range":"0-7fff",   "state":"active",   "replicas":{"core_node62":{ 
  "core":"backuprestore_restored_shard2_replica_n61",   
"base_url":"http://127.0.0.1:65140/solr;,   
"node_name":"127.0.0.1:65140_solr",   "state":"active",   
"type":"NRT",   "force_set_state":"false",   "leader":"true"}}, 
  "stateTimestamp":"1542804977102376356"}, "shard1_1":{   
"range":"c000-",   "state":"active",   
"replicas":{"core_node64":{   
"core":"backuprestore_restored_shard1_1_replica_n63",   
"base_url":"http://127.0.0.1:65140/solr;,   
"node_name":"127.0.0.1:65140_solr",   "state":"active",   
"type":"NRT",   "force_set_state":"false",   "leader":"true"}}, 
  "stateTimestamp":"1542804977102438283"}, "shard1_0":{   
"range":"8000-bfff",   "state":"active",   
"replicas":{"core_node66":{   
"core":"backuprestore_restored_shard1_0_replica_n65",   
"base_url":"http://127.0.0.1:65140/solr;,   
"node_name":"127.0.0.1:65140_solr",   "state":"active",   
"type":"NRT",   "force_set_state":"false",   "leader":"true"}}, 
  "stateTimestamp":"1542804977102516030"}},   "router":{ 
"name":"compositeId", "field":"shard_s"},   "maxShardsPerNode":"2",   
"autoAddReplicas":"true",   "nrtReplicas":1,   "tlogReplicas":0}

Stack Trace:
java.lang.AssertionError: Node 127.0.0.1:65140_solr has 3 replicas. Expected 
num replicas : 2. state: 
DocCollection(backuprestore_restored//collections/backuprestore_restored/state.json/11)={
  "pullReplicas":0,
  "replicationFactor":1,
  "shards":{
"shard2":{
  "range":"0-7fff",
  "state":"active",
  "replicas":{"core_node62":{
  "core":"backuprestore_restored_shard2_replica_n61",
  "base_url":"http://127.0.0.1:65140/solr;,
  "node_name":"127.0.0.1:65140_solr",
  "state":"active",
  "type":"NRT",
  "force_set_state":"false",
  "leader":"true"}},
  "stateTimestamp":"1542804977102376356"},
"shard1_1":{
  "range":"c000-",
  "state":"active",
  "replicas":{"core_node64":{
  "core":"backuprestore_restored_shard1_1_replica_n63",
  "base_url":"http://127.0.0.1:65140/solr;,
  "node_name":"127.0.0.1:65140_solr",
  "state":"active",
  "type":"NRT",
  "force_set_state":"false",
  "leader":"true"}},
  "stateTimestamp":"1542804977102438283"},
"shard1_0":{
  "range":"8000-bfff",
  "state":"active",
  "replicas":{"core_node66":{
  "core":"backuprestore_restored_shard1_0_replica_n65",
  "base_url":"http://127.0.0.1:65140/solr;,
  "node_name":"127.0.0.1:65140_solr",
  "state":"active",
  "type":"NRT",
  "force_set_state":"false",
  "leader":"true"}},
  "stateTimestamp":"1542804977102516030"}},
  "router":{
"name":"compositeId",
"field":"shard_s"},
  "maxShardsPerNode":"2",
  "autoAddReplicas":"true",
  "nrtReplicas":1,
  "tlogReplicas":0}
at 
__randomizedtesting.SeedInfo.seed([DFD90DF4AD7ED7E1:578D322E0382BA19]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.lambda$testBackupAndRestore$1(AbstractCloudBackupRestoreTestCase.java:339)
at java.util.HashMap.forEach(HashMap.java:1289)
at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.testBackupAndRestore(AbstractCloudBackupRestoreTestCase.java:338)
at 
org.apache.solr.cloud.api.collections.AbstractCloudBackupRestoreTestCase.test(AbstractCloudBackupRestoreTestCase.java:144)
at 
org.apache.solr.cloud.api.collections.TestLocalFSCloudBackupRestore.test(TestLocalFSCloudBackupRestore.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 

[GitHub] lucene-solr issue #494: ant compile error

[diegoceccarelli]

Github user diegoceccarelli commented on the issue:

https://github.com/apache/lucene-solr/pull/494
  
`maven.restlet.org` gets redirected to `maven.restlet.com`, LGTM


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8572) StringIndexOutOfBoundsException in parser/EscapeQuerySyntaxImpl.java

[Octavian Mocanu (JIRA)]

Octavian Mocanu created LUCENE-8572:
---

 Summary: StringIndexOutOfBoundsException in 
parser/EscapeQuerySyntaxImpl.java
 Key: LUCENE-8572
 URL: https://issues.apache.org/jira/browse/LUCENE-8572
 Project: Lucene - Core
  Issue Type: Bug
  Components: core/queryparser
Affects Versions: 6.3
Reporter: Octavian Mocanu


With "lucene-queryparser-6.3.0", specifically in
"org/apache/lucene/queryparser/flexible/standard/parser/EscapeQuerySyntaxImpl.java"
 
when escaping strings containing extended unicode chars, and with a locale 
distinct from that of the character set the string uses, the process fails, 
with a "java.lang.StringIndexOutOfBoundsException".
 
The reason is that the comparison is done by previously converting all of the 
characters of the string to lower case chars, and by doing this, the original 
string size isn't anymore the same, but less, as of the transformed one, so 
that executing
 
org/apache/lucene/queryparser/flexible/standard/parser/EscapeQuerySyntaxImpl.java:89

fails with a java.lang.StringIndexOutOfBoundsException.

I wonder whether the transformation to lower case is really needed when 
treating the escape chars, since by avoiding it, the error may be avoided.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-7.x-Linux (64bit/jdk-10.0.1) - Build # 123 - Still unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-7.x-Linux/123/
Java: 64bit/jdk-10.0.1 -XX:-UseCompressedOops -XX:+UseParallelGC

7 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:37391/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:37391/collection1
at 
__randomizedtesting.SeedInfo.seed([41026570928505AF:C9565AAA3C796857]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:509)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:498)
at 
org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:176)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-12-ea+12) - Build # 23246 - Still unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23246/
Java: 64bit/jdk-12-ea+12 -XX:-UseCompressedOops -XX:+UseSerialGC

91 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.schema.SchemaApiFailureTest

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
at __randomizedtesting.SeedInfo.seed([A34808AA381DBDD9]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1474)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:450)
at 
org.apache.solr.schema.SchemaApiFailureTest.setupCluster(SchemaApiFailureTest.java:43)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.SchemaApiFailureTest

Error Message:


Stack Trace:
java.util.concurrent.TimeoutException
at __randomizedtesting.SeedInfo.seed([A34808AA381DBDD9]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.waitForState(ZkStateReader.java:1474)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.waitForState(CloudSolrClient.java:450)
at 
org.apache.solr.schema.SchemaApiFailureTest.setupCluster(SchemaApiFailureTest.java:43)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[JENKINS] Lucene-Solr-7.x-Windows (64bit/jdk1.8.0_172) - Build # 896 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/896/
Java: 64bit/jdk1.8.0_172 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

18 tests failed.
FAILED:  
org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd.testFullImportBadConfig

Error Message:
Could not remove the following files (in the order of attempts):
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002\collection1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002\collection1

C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002
 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002\collection1:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002\collection1
   
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\contrib\solr-dataimporthandler\test\J1\temp\solr.handler.dataimport.TestSolrEntityProcessorEndToEnd_E1F080AF3D169482-001\tempDir-002

at 
__randomizedtesting.SeedInfo.seed([E1F080AF3D169482:AE86562C759A0F69]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:318)
at 
org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd$SolrInstance.tearDown(TestSolrEntityProcessorEndToEnd.java:361)
at 
org.apache.solr.handler.dataimport.TestSolrEntityProcessorEndToEnd.tearDown(TestSolrEntityProcessorEndToEnd.java:142)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:993)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 

[jira] [Commented] (SOLR-12727) Upgrade ZooKeeper dependency to 3.4.13

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695530#comment-16695530
 ] 

Erick Erickson commented on SOLR-12727:
---

[~risdenk] I'm back looking at this. I'm not sure where to add
{code:java}
permission javax.security.auth.kerberos.ServicePermission 
"zookeeper/localh...@example.com", "initiate";
permission javax.security.auth.kerberos.ServicePermission 
"zookeeper/localh...@example.com", "accept";
{code}

Understand I'm a total and complete Kerberos novice. I've @Ignored that test 
for now so I can get through the rest of the tests. I had a million lines or so 
of: 

1963190 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [] 
o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.lang.NullPointerException: null
   [junit4]   2> 1963190 WARN  (NIOServerCxn.Factory:0.0.0.0/0.0.0.0:0) [] 
o.a.z.s.NIOServerCnxnFactory Ignoring unexpected runtime exception
   [junit4]   2> java.lang.NullPointerException: null

> Upgrade ZooKeeper dependency to 3.4.13
> --
>
> Key: SOLR-12727
> URL: https://issues.apache.org/jira/browse/SOLR-12727
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud
>Affects Versions: 7.4
>Reporter: Shawn Heisey
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12727.patch
>
>
> Upgrade ZK dependency to 3.4.13.  This fixes ZOOKEEPER-2184 which will make 
> the ZK client re-resolve the server hostnames when a connection fails.  This 
> will fix issues where a failed ZK container is replaced with a new one that 
> has a different IP address and DNS gets updated with the new address.
> Typically these upgrades do not require code changes, but that should be 
> verified.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[mosh (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695587#comment-16695587
 ] 

mosh commented on SOLR-5211:


{quote}[junit4] - org.apache.solr.schema.MultiTermTest.testQueryCopiedToMulti
 [junit4] - org.apache.solr.schema.MultiTermTest.testDefaultCopiedToMulti
{quote}
I could not reproduce the failures in these tests.
 Were the exceptions reproducible using a particular seed?
 I ran the test suite using this command:
{code}
ant clean test -Dtestcase=MultiTermTest{code}

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 1986 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/1986/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.6/8/consoleText

[repro] Revision: bbe9511894340df14b938565cf3575540ca2f21b

[repro] Ant options: -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  -Dtestcase=RestartWhileUpdatingTest 
-Dtests.method=test -Dtests.seed=5316C9DAF434123D -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.locale=tr-TR -Dtests.timezone=Etc/GMT+5 -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[repro] Repro line:  ant test  -Dtestcase=CloudSolrClientTest 
-Dtests.method=testRouting -Dtests.seed=90E7D12003CB94EA -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.locale=ms-MY -Dtests.timezone=America/Atka -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
67cdd21996f716ffb137bbcb8f826794a2632be7
[repro] git fetch
[repro] git checkout bbe9511894340df14b938565cf3575540ca2f21b

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   RestartWhileUpdatingTest
[repro]solr/solrj
[repro]   CloudSolrClientTest
[repro] ant compile-test

[...truncated 3580 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.RestartWhileUpdatingTest" -Dtests.showOutput=onerror 
-Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.seed=5316C9DAF434123D -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.locale=tr-TR -Dtests.timezone=Etc/GMT+5 -Dtests.asserts=true 
-Dtests.file.encoding=UTF-8

[...truncated 37596 lines...]
[repro] Setting last failure code to 256

[repro] ant compile-test

[...truncated 454 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.CloudSolrClientTest" -Dtests.showOutput=onerror 
-Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.seed=90E7D12003CB94EA -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-7.6/test-data/enwiki.random.lines.txt
 -Dtests.locale=ms-MY -Dtests.timezone=America/Atka -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 143 lines...]
[repro] Failures:
[repro]   0/5 failed: org.apache.solr.client.solrj.impl.CloudSolrClientTest
[repro]   1/5 failed: org.apache.solr.cloud.RestartWhileUpdatingTest
[repro] git checkout 67cdd21996f716ffb137bbcb8f826794a2632be7

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: 2 Level grouping (solr 5.5.3)

[Ian Caldwell]

Shawn I have added a patch to SOLR-13005 as it was a small change.

The problem with SOLR-13004 Integer overflow is that in extending the code for 
the second level of grouping I was trying not to change code that I didn't need 
to be change or would impact into other parts of the system(trying to keep the 
changes isolated to grouping where possible). The totals(when grouping) are 
held in the ResponseBuilder and I was not changing it and able to hold and 
accumulate the totals in my new grouping code. I am not sure how much work 
would be involved with changing to a long, but I am busy at the moment but may 
be able to look at a patch later.


Ian
NLA

ResponseBuilder line 184
  public int totalHitCount; // Hit count used when distributed grouping is 
performed.


-Original Message-
From: Shawn Heisey  
Sent: Wednesday, 21 November 2018 3:58 PM
To: dev@lucene.apache.org
Subject: Re: 2 Level grouping (solr 5.5.3)

On 11/20/2018 9:45 PM, Ian Caldwell wrote:
>
> Yes the bugs were in the std code JIRS created.
>
> https://issues.apache.org/jira/browse/SOLR-13004
>
> https://issues.apache.org/jira/browse/SOLR-13005
>

Can you attach patches to the issues that fix the problems?  We will need to 
confirm whether the problems you found are still there in current versions -- 
5.5.3 is more than a little outdated now.  It is two major versions back from 
current, and was released two years ago.  In open source terms, that's ancient. 
 Someone else may have noticed the problems and fixed them already.

Thanks,
Shawn


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional 
commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-7.6-Linux (64bit/jdk-12-ea+12) - Build # 13 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.6-Linux/13/
Java: 64bit/jdk-12-ea+12 -XX:-UseCompressedOops -XX:+UseG1GC

17 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.client.solrj.request.TestV2Request: 1) Thread[id=502, 
name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request] 
at java.base@12-ea/java.lang.Thread.sleep(Native Method) at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.base@12-ea/java.lang.Thread.run(Thread.java:835)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.client.solrj.request.TestV2Request: 
   1) Thread[id=502, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestV2Request]
at java.base@12-ea/java.lang.Thread.sleep(Native Method)
at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
at __randomizedtesting.SeedInfo.seed([E8B48F2704282DA1]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.request.TestV2Request

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.client.solrj.request.TestV2Request: 1) Thread[id=2610, 
name=Connection evictor, state=TIMED_WAITING, group=TGRP-TestV2Request] 
at java.base@12-ea/java.lang.Thread.sleep(Native Method) at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
 at java.base@12-ea/java.lang.Thread.run(Thread.java:835)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.client.solrj.request.TestV2Request: 
   1) Thread[id=2610, name=Connection evictor, state=TIMED_WAITING, 
group=TGRP-TestV2Request]
at java.base@12-ea/java.lang.Thread.sleep(Native Method)
at 
app//org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
at java.base@12-ea/java.lang.Thread.run(Thread.java:835)
at __randomizedtesting.SeedInfo.seed([E8B48F2704282DA1]:0)


FAILED:  org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient

Error Message:
Error from server at https://127.0.0.1:39511/solr: no such collection or alias

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException: 
Error from server at https://127.0.0.1:39511/solr: no such collection or alias
at 
__randomizedtesting.SeedInfo.seed([E8B48F2704282DA1:30AC970DA51C3F86]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:830)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260)
at 
org.apache.solr.client.solrj.request.TestV2Request.assertSuccess(TestV2Request.java:49)
at 
org.apache.solr.client.solrj.request.TestV2Request.doTest(TestV2Request.java:96)
at 
org.apache.solr.client.solrj.request.TestV2Request.testHttpSolrClient(TestV2Request.java:70)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1703 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1703/

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds

Error Message:
expected: but was:

Stack Trace:
java.lang.AssertionError: expected: but was:
at 
__randomizedtesting.SeedInfo.seed([E360836268B3:E9F4C8CDCED963E9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:147)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMixedBounds(IndexSizeTriggerTest.java:673)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)




Build Log:
[...truncated 14957 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest
   [junit4]   2> Creating dataDir: 

[JENKINS] Lucene-Solr-BadApples-Tests-master - Build # 216 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-Tests-master/216/

3 tests failed.
FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in time
at 
__randomizedtesting.SeedInfo.seed([39C0A1E2309D537:3A12B35E0CF61CC9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testSplitIntegration(IndexSizeTriggerTest.java:320)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.sim.TestSimTriggerIntegration.testNodeMarkersRegistration

Error Message:
Path /autoscaling/nodeAdded/127.0.0.1:10203_solr should have been deleted

Stack Trace:
java.lang.AssertionError: Path /autoscaling/nodeAdded/127.0.0.1:10203_solr 
should have been deleted

[jira] [Updated] (LUCENE-8570) Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'

[YOO JEONGIN (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YOO JEONGIN updated LUCENE-8570:

Description: 
안녕하세요 
저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
"Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
고맙습니다.

!image-2018-11-21-15-45-57-644.png!

!image-2018-11-21-15-46-28-114.png!

!image-2018-11-21-15-46-45-081.png!

  !image-2018-11-22-11-36-09-190.png!

  was:
Hello
I have a question about applying synonyms to korean morphological analyzer 
'Nori'.
If you perform a synonym filter after tokenizing with 'nori', there is no 
response from the analyzer as shown below. I want to know if this is a bug, not 
in the original function, or if I made the wrong setting.
If you know the correct way to apply a synonym filter in "Nori," I'd like to 
ask you for your advice.
Thank you.

!image-2018-11-21-15-45-57-644.png!

!image-2018-11-21-15-46-28-114.png!

!image-2018-11-21-15-46-45-081.png!

 


> Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'
> ---
>
> Key: LUCENE-8570
> URL: https://issues.apache.org/jira/browse/LUCENE-8570
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 7.5
>Reporter: YOO JEONGIN
>Priority: Major
> Attachments: image-2018-11-21-15-45-57-644.png, 
> image-2018-11-21-15-46-28-114.png, image-2018-11-21-15-46-45-081.png, 
> image-2018-11-22-11-36-09-190.png, image-2018-11-22-11-36-27-916.png
>
>
> 안녕하세요 
> 저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
> 아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
> "Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
> 고맙습니다.
> !image-2018-11-21-15-45-57-644.png!
> !image-2018-11-21-15-46-28-114.png!
> !image-2018-11-21-15-46-45-081.png!
>   !image-2018-11-22-11-36-09-190.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8570) Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'

[YOO JEONGIN (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YOO JEONGIN updated LUCENE-8570:

Attachment: image-2018-11-22-11-36-09-190.png

> Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'
> ---
>
> Key: LUCENE-8570
> URL: https://issues.apache.org/jira/browse/LUCENE-8570
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 7.5
>Reporter: YOO JEONGIN
>Priority: Major
> Attachments: image-2018-11-21-15-45-57-644.png, 
> image-2018-11-21-15-46-28-114.png, image-2018-11-21-15-46-45-081.png, 
> image-2018-11-22-11-36-09-190.png, image-2018-11-22-11-36-27-916.png
>
>
> 안녕하세요 
> 저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
> 아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
> "Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
> 고맙습니다.
> !image-2018-11-21-15-45-57-644.png!
> !image-2018-11-21-15-46-28-114.png!
> !image-2018-11-21-15-46-45-081.png!
>   !image-2018-11-22-11-36-09-190.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.4) - Build # 3129 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3129/
Java: 64bit/jdk-9.0.4 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

5 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:36515/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:36515/solr
at 
__randomizedtesting.SeedInfo.seed([8F37E627BC1B3000:4EC79F8B914BFAA7]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:35285/solr

Stack Trace:

[GitHub] lucene-solr pull request #501: SOLR-5211

[moshebla]

Github user moshebla commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/501#discussion_r235613814
  
--- Diff: 
solr/core/src/test/org/apache/solr/cloud/TestCloudPseudoReturnFields.java ---
@@ -194,7 +194,7 @@ public void testAllRealFields() throws Exception {
   SolrDocumentList docs = assertSearch(params("q", "*:*", "rows", 
"10", "fl",fl));
   // shouldn't matter what doc we pick...
   for (SolrDocument doc : docs) {
-assertEquals(fl + " => " + doc, 4, doc.size());
+assertEquals(fl + " => " + doc, 5, doc.size());
--- End diff --

Doc size increased by 1 since _root_ is also returned by queries.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 4940 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4940/
Java: 64bit/jdk-9 -XX:-UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 14783 lines...]
   [junit4] JVM J1: stdout was not empty, see: 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp/junit4-J1-20181122_034325_5789267970167709730985.sysout
   [junit4] >>> JVM J1 emitted unexpected output (verbatim) 
   [junit4] #
   [junit4] # A fatal error has been detected by the Java Runtime Environment:
   [junit4] #
   [junit4] #  SIGSEGV (0xb) at pc=0x0001032f49d8, pid=52634, tid=17155
   [junit4] #
   [junit4] # JRE version: Java(TM) SE Runtime Environment (9.0+181) (build 
9+181)
   [junit4] # Java VM: Java HotSpot(TM) 64-Bit Server VM (9+181, mixed mode, 
tiered, parallel gc, bsd-amd64)
   [junit4] # Problematic frame:
   [junit4] # V  [libjvm.dylib+0x4f49d8]  Node::in(unsigned int) const+0x18
   [junit4] #
   [junit4] # No core dump will be written. Core dumps have been disabled. To 
enable core dumping, try "ulimit -c unlimited" before starting Java again
   [junit4] #
   [junit4] # An error report file with more information is saved as:
   [junit4] # 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/hs_err_pid52634.log
   [junit4] #
   [junit4] # Compiler replay data is saved as:
   [junit4] # 
/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1/replay_pid52634.log
   [junit4] #
   [junit4] # If you would like to submit a bug report, please visit:
   [junit4] #   http://bugreport.java.com/bugreport/crash.jsp
   [junit4] #
   [junit4] <<< JVM J1: EOF 

[...truncated 696 lines...]
   [junit4] ERROR: JVM J1 ended with an exception, command line: 
/Library/Java/JavaVirtualMachines/jdk-9.jdk/Contents/Home/bin/java 
-XX:-UseCompressedOops -XX:+UseParallelGC -XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/heapdumps 
-ea -esa --illegal-access=deny -Dtests.prefix=tests 
-Dtests.seed=EFFCF65B3CED25E7 -Xmx512M -Dtests.iters= -Dtests.verbose=false 
-Dtests.infostream=false -Dtests.codec=random -Dtests.postingsformat=random 
-Dtests.docvaluesformat=random -Dtests.locale=random -Dtests.timezone=random 
-Dtests.directory=random -Dtests.linedocsfile=europarl.lines.txt.gz 
-Dtests.luceneMatchVersion=8.0.0 -Dtests.cleanthreads=perClass 
-Djava.util.logging.config.file=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/tools/junit4/logging.properties
 -Dtests.nightly=false -Dtests.weekly=false -Dtests.monster=false 
-Dtests.slow=true -Dtests.asserts=true -Dtests.multiplier=1 -DtempDir=./temp 
-Djava.io.tmpdir=./temp 
-Dcommon.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene 
-Dclover.db.dir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/build/clover/db
 
-Djava.security.policy=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/lucene/tools/junit4/solr-tests.policy
 -Dtests.LUCENE_VERSION=8.0.0 -Djetty.testMode=1 -Djetty.insecurerandom=1 
-Dsolr.directoryFactory=org.apache.solr.core.MockDirectoryFactory 
-Djava.awt.headless=true -Djdk.map.althashing.threshold=0 
-Dtests.src.home=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX 
-Djava.security.egd=file:/dev/./urandom 
-Djunit4.childvm.cwd=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/J1
 
-Djunit4.tempDir=/Users/jenkins/workspace/Lucene-Solr-master-MacOSX/solr/build/solr-core/test/temp
 -Djunit4.childvm.id=1 -Djunit4.childvm.count=2 -Dtests.disableHdfs=true 
-Djava.security.manager=org.apache.lucene.util.TestSecurityManager 
-Dtests.filterstacks=true -Dtests.leaveTemporary=false -Dtests.badapples=false 
-classpath 

[jira] [Updated] (LUCENE-8570) Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'

[YOO JEONGIN (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YOO JEONGIN updated LUCENE-8570:

Attachment: image-2018-11-22-11-36-27-916.png

> Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'
> ---
>
> Key: LUCENE-8570
> URL: https://issues.apache.org/jira/browse/LUCENE-8570
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 7.5
>Reporter: YOO JEONGIN
>Priority: Major
> Attachments: image-2018-11-21-15-45-57-644.png, 
> image-2018-11-21-15-46-28-114.png, image-2018-11-21-15-46-45-081.png, 
> image-2018-11-22-11-36-09-190.png, image-2018-11-22-11-36-27-916.png
>
>
> 안녕하세요 
>  저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
>  아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
>  "Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
>  고맙습니다.
> !image-2018-11-21-15-45-57-644.png!
> !image-2018-11-21-15-46-28-114.png!
> !image-2018-11-21-15-46-45-081.png!
> !image-2018-11-22-11-36-27-916.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8570) Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'

[YOO JEONGIN (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

YOO JEONGIN updated LUCENE-8570:

Description: 
안녕하세요 
 저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
 아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
 "Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
 고맙습니다.

!image-2018-11-21-15-45-57-644.png!

!image-2018-11-21-15-46-28-114.png!

!image-2018-11-21-15-46-45-081.png!

!image-2018-11-22-11-36-27-916.png!

  was:
안녕하세요 
저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
"Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
고맙습니다.

!image-2018-11-21-15-45-57-644.png!

!image-2018-11-21-15-46-28-114.png!

!image-2018-11-21-15-46-45-081.png!

  !image-2018-11-22-11-36-09-190.png!


> Issue that the synonym filter is not executed in the KOREAN analyzer 'Nori'
> ---
>
> Key: LUCENE-8570
> URL: https://issues.apache.org/jira/browse/LUCENE-8570
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/analysis
>Affects Versions: 7.5
>Reporter: YOO JEONGIN
>Priority: Major
> Attachments: image-2018-11-21-15-45-57-644.png, 
> image-2018-11-21-15-46-28-114.png, image-2018-11-21-15-46-45-081.png, 
> image-2018-11-22-11-36-09-190.png, image-2018-11-22-11-36-27-916.png
>
>
> 안녕하세요 
>  저는 한국형 형태 분석입니다. 'Nori'를 사용하여 적용 가능한 질문에 답하십시오. 
>  아직까지는 필터가 작동하지 않아 응답이 없습니다. 이 버그는 본래의 기능과 다를 수 있습니다. 
>  "Nori"를 적용 할 수있는 적절한 방법을 선택하십시오. 
>  고맙습니다.
> !image-2018-11-21-15-45-57-644.png!
> !image-2018-11-21-15-46-28-114.png!
> !image-2018-11-21-15-46-45-081.png!
> !image-2018-11-22-11-36-27-916.png!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-master-Linux (64bit/jdk1.8.0_172) - Build # 124 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-BadApples-master-Linux/124/
Java: 64bit/jdk1.8.0_172 -XX:-UseCompressedOops -XX:+UseParallelGC

4 tests failed.
FAILED:  org.apache.solr.cloud.BasicDistributedZkTest.test

Error Message:
Timeout occured while waiting response from server at: 
http://127.0.0.1:38499/collection1

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:38499/collection1
at 
__randomizedtesting.SeedInfo.seed([B36B4D1C26E8E763:3B3F72C688148A9B]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:654)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:509)
at 
org.apache.solr.BaseDistributedSearchTestCase.indexDoc(BaseDistributedSearchTestCase.java:498)
at 
org.apache.solr.cloud.BasicDistributedZkTest.test(BasicDistributedZkTest.java:176)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1010)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-9379) Need cloud & RTG testing of [child] transformer

[mosh (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695593#comment-16695593
 ] 

mosh commented on SOLR-9379:


SOLR-12638 uses RTG and ChildDocTransformer in its tests, including its cloud 
based tests.

> Need cloud & RTG testing of [child] transformer
> ---
>
> Key: SOLR-9379
> URL: https://issues.apache.org/jira/browse/SOLR-9379
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Priority: Major
>
> Spun off of SOLR-9314...
> We don't seem to have any cloud based tests of the {{\[child\]}} 
> DocTransformer, or any tests of using it with RTG (on either committed or 
> uncommited docs).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13007) Use javabin instead of JSON to send messages to overseer

[Noble Paul (JIRA)]

Noble Paul created SOLR-13007:
-

 Summary: Use javabin instead of JSON to send messages to overseer
 Key: SOLR-13007
 URL: https://issues.apache.org/jira/browse/SOLR-13007
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Noble Paul


The messages themselves are ephemeral and the readability is not a big issue. 
Using javabin can:
* reduce the payload size
* make processing faster




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Windows (64bit/jdk-12-ea+12) - Build # 7631 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7631/
Java: 64bit/jdk-12-ea+12 -XX:-UseCompressedOops -XX:+UseSerialGC

21 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Expected new active leader null Live Nodes: [127.0.0.1:60949_solr, 
127.0.0.1:60950_solr, 127.0.0.1:60955_solr] Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/11)={
   "pullReplicas":"0",   "replicationFactor":"2",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node4":{   "core":"raceDeleteReplica_true_shard1_replica_n2", 
  "base_url":"http://127.0.0.1:60958/solr;,   
"node_name":"127.0.0.1:60958_solr",   "state":"down",   
"type":"NRT",   "leader":"true"}, "core_node6":{   
"core":"raceDeleteReplica_true_shard1_replica_n5",   
"base_url":"http://127.0.0.1:60958/solr;,   
"node_name":"127.0.0.1:60958_solr",   "state":"down",   
"type":"NRT"}, "core_node3":{   
"core":"raceDeleteReplica_true_shard1_replica_n1",   
"base_url":"http://127.0.0.1:60949/solr;,   
"node_name":"127.0.0.1:60949_solr",   "state":"down",   
"type":"NRT",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",  
 "autoAddReplicas":"false",   "nrtReplicas":"2",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Expected new active leader
null
Live Nodes: [127.0.0.1:60949_solr, 127.0.0.1:60950_solr, 127.0.0.1:60955_solr]
Last available state: 
DocCollection(raceDeleteReplica_true//collections/raceDeleteReplica_true/state.json/11)={
  "pullReplicas":"0",
  "replicationFactor":"2",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node4":{
  "core":"raceDeleteReplica_true_shard1_replica_n2",
  "base_url":"http://127.0.0.1:60958/solr;,
  "node_name":"127.0.0.1:60958_solr",
  "state":"down",
  "type":"NRT",
  "leader":"true"},
"core_node6":{
  "core":"raceDeleteReplica_true_shard1_replica_n5",
  "base_url":"http://127.0.0.1:60958/solr;,
  "node_name":"127.0.0.1:60958_solr",
  "state":"down",
  "type":"NRT"},
"core_node3":{
  "core":"raceDeleteReplica_true_shard1_replica_n1",
  "base_url":"http://127.0.0.1:60949/solr;,
  "node_name":"127.0.0.1:60949_solr",
  "state":"down",
  "type":"NRT",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"2",
  "tlogReplicas":"0"}
at 
__randomizedtesting.SeedInfo.seed([9A01175D5B02E56B:F017768D33F0AFA1]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:280)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:328)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:223)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

Re: Lucene/Solr 7.6

[Andrzej Białecki]

Hi Nicholas,

If it’s ok I would like to merge a small fix to the Ref Guide, spotted by 
Christine in SOLR-9856.

> On 1 Nov 2018, at 21:38, Nicholas Knize  wrote:
> 
> Hi all,
> 
> To follow up from our discussion on the 8.0 thread, I would like to cut the 
> 7.6 branch on either Tuesday or Wednesday of next week. Since this implies 
> feature freeze I went ahead and had a look at some of the issues that are 
> labeled for 7.6.
> 
> It looks like we only have one active issue listed as a blocker for Solr. The 
> upgrade notes in SOLR-12927 
> 
> For Lucene we have five active issues (each with a patch provided) listed as 
> blockers targeted for 7.6.
> 
> If there are any other issues that need to land before cutting the branch, 
> and they are not already labeled, please either mark them as blockers 
> accordingly or let me know prior to cutting the branch next Tuesday or 
> Wednesday.
> 
> Thank you!
> 
> - Nick
> -- 
> Nicholas Knize, Ph.D., GISP
> Geospatial Software Guy  |  Elasticsearch
> Apache Lucene Committer
> nkn...@apache.org   
> 



—

Andrzej Białecki

[jira] [Commented] (SOLR-9856) Collect metrics for shard replication and tlog replay on replicas

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695053#comment-16695053
 ] 

ASF subversion and git services commented on SOLR-9856:
---

Commit 67cdd21996f716ffb137bbcb8f826794a2632be7 in lucene-solr's branch 
refs/heads/master from Andrzej Bialecki
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=67cdd21 ]

SOLR-9856: Update the Ref Guide to no longer list this issue as unresolved.


> Collect metrics for shard replication and tlog replay on replicas
> -
>
> Key: SOLR-9856
> URL: https://issues.apache.org/jira/browse/SOLR-9856
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.0
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: 6.4, 7.0
>
> Attachments: SOLR-9856.patch
>
>
> Using API from SOLR-4735 add metrics for tracking outgoing replication from 
> leader to shard replicas, and for tracking transaction log processing on 
> replicas.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9856) Collect metrics for shard replication and tlog replay on replicas

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16695055#comment-16695055
 ] 

ASF subversion and git services commented on SOLR-9856:
---

Commit 6dbdd643c989dc82af5bf0a1f8b5b991c0db090f in lucene-solr's branch 
refs/heads/branch_7x from Andrzej Bialecki
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6dbdd64 ]

SOLR-9856: Update the Ref Guide to no longer list this issue as unresolved.


> Collect metrics for shard replication and tlog replay on replicas
> -
>
> Key: SOLR-9856
> URL: https://issues.apache.org/jira/browse/SOLR-9856
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: metrics
>Affects Versions: 7.0
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Minor
> Fix For: 6.4, 7.0
>
> Attachments: SOLR-9856.patch
>
>
> Using API from SOLR-4735 add metrics for tracking outgoing replication from 
> leader to shard replicas, and for tracking transaction log processing on 
> replicas.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694919#comment-16694919
 ] 

David Smiley commented on SOLR-5211:


[~moshebla] there are still some test failures:
   [junit4]   - 
org.apache.solr.cloud.TestCloudPseudoReturnFields.testScoreAndAllRealFieldsRTG
   [junit4]   - 
org.apache.solr.cloud.TestCloudPseudoReturnFields.testAllRealFieldsRTG
   [junit4]   - 
org.apache.solr.cloud.TestCloudPseudoReturnFields.testAllRealFields
   [junit4]   - 
org.apache.solr.cloud.TestCloudPseudoReturnFields.testScoreAndAllRealFields
   [junit4]   - org.apache.solr.RootFieldTest (suite)
   [junit4]   - org.apache.solr.schema.MultiTermTest.testQueryCopiedToMulti
   [junit4]   - org.apache.solr.schema.MultiTermTest.testDefaultCopiedToMulti

RootFieldTest complained schema-11.xml doesn't exist

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: SOLR: Unnecessary logging

[Erick Erickson]

bq. t is super easy to change the log level of one particular class or package

If anyone has prod logs laying around collected at INFO level and could
provide any counts of what packages the most frequent log messages come
from, please attach the data to SOLR-11934. I've seen some cases where
just changing the log levels for a single class from INFO to  WARN reduced
the log volume tremendously (90% or  so). That was an app that had  a
high update volume of a single doc at a time so doesn't reflect other
scenarios.

I'll be glad to analyze any shareable log files, but understand that sharing
log files often violates corporate policy.

My suspicion is that there are just a few classes that could be configured
at WARN level that would reduce the volume of log messages greatly.

That's really a separate question from what level we log messages at,
much of the logging in the code is specified at levels suitable for
troubleshooting rather than an ops perspective. But that's a discussion
for whenever I/we get to SOLR-11934. I think configuring  some
classes to WARN by default or just providing them commented-out
in the log4j2.xml files would be an easy win.

**

bq. We can find a way to distinguish between dev/prod
usage and only warn about file descriptors, suspiciously low heap,
lack of SSL etc if
Solr is bound to a public IP address.

Don't want to bikeshed this too much, so take this for what it's worth.

Seeing it for the first time when you deploy to prod is too late IMO. I
suppose we're looking through two different lenses. We bounce around
from client to client, so my lens is how often people new to
Solr and/or operations folks who get the finished app from development
miss things like this. And then spend time in panic mode because their
prod system broke. And how long we've spent on the phone with them
before remembering to check this (it's part of a standard checklist
now of course, but still)... And how it tarnishes their opinion of Solr.

I'd be less worried about this if we could throw errors that pointed to
this issue, but the errors that come out when ulimits are exceeded
are obscure and varied.

Sure, the same problem can occur if the devs set the variable
in solr.in.sh then forget about it and don't tell ops to bump these
limits, but at least some of the time the problem will  be avoided with
the message front and center.

Given the cost to devs is to set a single system var I think it's a price
worth paying. All IMO and reflecting my experience of course.

Best,
Erick
On Wed, Nov 21, 2018 at 12:40 AM Jan Høydahl  wrote:
>
> bq. I don't want to see file limit warnings on a dev laptop
>
> "If you no longer wish to see this warning, set SOLR_ULIMIT_CHECKS to
> false in your profile or solr.in.sh"
>
> Not having these limits set properly has caused weird errors for a lot
> of people, but I agree they are annoying at times so there's a way to
> shut them off.
>
>
> Note my wording - "on a dev laptop". We can find a way to distinguish between 
> dev/prod
> usage and only warn about file descriptors, suspiciously low heap, lack of 
> SSL etc if
> Solr is bound to a public IP address.
>
> The only logging config files you should have to worry about are in
> ./solr/server/resources.
>
> But yeah, there are several Solr JIRAs lying around that I'll break
> some time loose _sometime_ to work on  to try to make logging more
> rational. Unfortunately as Shawn says, there's always tension. When
> running "normally", I pretty much only want to see WARN messages. Then
> every time I want to diagnose something, I want to see _everything_.
> And telling the user to change their logging and call me again next
> time the problem happens is not very satisfying for anyone.
>
>
> It is super easy to change the log level of one particular class or package
> selectively, that's why we have log4j2.xml and also the Admin's Logging 
> levels menu.
> The occational need to see all of the 150 jar files loaded on startup should 
> not be
> solved by spewing out that for first-time users of Solr who really just care 
> for whether
> the server started OK or not.
>
> Agree that adding commented-out snippets in the logging config could be a
> good way to quickly let people debug "ClassNotFound issues", "Indexing 
> issues",
> "Faceting issues", "Security issues" etc. We could have pre-canned configs 
> that
> would upper the logging of classes of particular interest for each case.
>
> Jan

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12775) Add a deprecated implementation of LowerCaseTokenizer

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16694923#comment-16694923
 ] 

Alan Woodward commented on SOLR-12775:
--

> Solr should log a warning

Ah turns out it does, I was looking in the wrong place - it emits it when it's 
first loaded, not when the core is created.  I'll remove the specialised logger.

> Add a deprecated implementation of LowerCaseTokenizer
> -
>
> Key: SOLR-12775
> URL: https://issues.apache.org/jira/browse/SOLR-12775
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-12775.patch, SOLR-12775.patch, SOLR-12775.patch, 
> SOLR-12775.patch
>
>
> LUCENE-8498 will remove LowerCaseTokenizer and LowerCaseTokenizerFactory from 
> lucene 8.  To make upgrading from Solr 7.x to Solr 8 easier for users who 
> have schemas that use LowerCaseTokenizerFactory, we should add a deprecated 
> copy of the code to the {{org.apache.solr.analysis}} package.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12775) Add a deprecated implementation of LowerCaseTokenizer

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated SOLR-12775:
-
Attachment: SOLR-12775.patch

> Add a deprecated implementation of LowerCaseTokenizer
> -
>
> Key: SOLR-12775
> URL: https://issues.apache.org/jira/browse/SOLR-12775
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>Priority: Blocker
> Fix For: master (8.0)
>
> Attachments: SOLR-12775.patch, SOLR-12775.patch, SOLR-12775.patch, 
> SOLR-12775.patch, SOLR-12775.patch
>
>
> LUCENE-8498 will remove LowerCaseTokenizer and LowerCaseTokenizerFactory from 
> lucene 8.  To make upgrading from Solr 7.x to Solr 8 easier for users who 
> have schemas that use LowerCaseTokenizerFactory, we should add a deprecated 
> copy of the code to the {{org.apache.solr.analysis}} package.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-10.0.1) - Build # 3127 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3127/
Java: 64bit/jdk-10.0.1 -XX:+UseCompressedOops -XX:+UseG1GC

4 tests failed.
FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:32913/solr

Stack Trace:
java.lang.AssertionError: Can not find doc 7 in https://127.0.0.1:32913/solr
at 
__randomizedtesting.SeedInfo.seed([D0AEF8A6F925360E:115E810AD475FCA9]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.TestTlogReplica.checkRTG(TestTlogReplica.java:902)
at 
org.apache.solr.cloud.TestTlogReplica.testRecovery(TestTlogReplica.java:567)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1742)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:935)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:971)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:985)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:944)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:830)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:891)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.solr.cloud.TestTlogReplica.testRecovery

Error Message:
Can not find doc 7 in https://127.0.0.1:35151/solr

Stack Trace: