[jira] [Commented] (LUCENE-8857) Refactor TopDocs#Merge To Take In Custom Tie Breakers

[Simon Willnauer (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8857?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866253#comment-16866253
 ] 

Simon Willnauer commented on LUCENE-8857:
-

>From my perspective we should simplify this even more and remove 
>_TieBreakingParameters_. TopDocs can use _Comparator_  and default 
>to the shard index if it's not supplied. That should be sufficient?

> Refactor TopDocs#Merge To Take In Custom Tie Breakers
> -
>
> Key: LUCENE-8857
> URL: https://issues.apache.org/jira/browse/LUCENE-8857
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Attachments: LUCENE-8857.patch, LUCENE-8857.patch
>
>
> In LUCENE-8829, the idea of having lambdas passed in to the API to allow 
> finer control over the process was discussed.
> This JIRA tracks adding a parameter to the API which allows passing in 
> lambdas to define custom tie breakers, thus allowing users to do custom 
> algorithms when required.
> CC: [~jpountz]  [~simonw] 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] s1monw commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

s1monw commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294611420
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -639,13 +639,15 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length - 1) { // execute the last on the caller 
thread
 
 Review comment:
   I pushed new changes


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866233#comment-16866233
 ] 

Lucene/Solr QA commented on SOLR-7530:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  4m 
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  4m 30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  4m 25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  4m 25s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 92m 
16s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  1m 
35s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 16s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-7530 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12972018/SOLR-7530.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene2-us-west.apache.org 4.4.0-112-generic #135-Ubuntu SMP 
Fri Jan 19 11:48:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 3030ea9 |
| ant | version: Apache Ant(TM) version 1.9.6 compiled on July 20 2018 |
| Default Java | LTS |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/442/testReport/ |
| modules | C: solr solr/core solr/test-framework U: solr |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/442/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[JENKINS] Lucene-Solr-NightlyTests-8.x - Build # 125 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-8.x/125/

7 tests failed.
FAILED:  org.apache.solr.cloud.LeaderVoteWaitTimeoutTest.basicTest

Error Message:
Timeout occurred while waiting response from server at: 
https://127.0.0.1:37171/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while 
waiting response from server at: https://127.0.0.1:37171/solr
at 
__randomizedtesting.SeedInfo.seed([310E25E3F2552304:C3FA3281B6F02E37]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:667)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1128)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:897)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:829)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:228)
at 
org.apache.solr.cloud.LeaderVoteWaitTimeoutTest.basicTest(LeaderVoteWaitTimeoutTest.java:155)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866179#comment-16866179
 ] 

Munendra S N commented on SOLR-7530:


[~mkhludnev]
Current changes entry looks great.  Thanks a lot for handling this

> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (PYLUCENE-50) StoredField of an int has the wrong type.

[Andi Vajda (JIRA)]

[ 
https://issues.apache.org/jira/browse/PYLUCENE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866175#comment-16866175
 ] 

Andi Vajda commented on PYLUCENE-50:


Thank you, Aric, for you reporting this.

Andi..


?? {{field = document.StoredField('', 0)}}


> StoredField of an int has the wrong type.
> -
>
> Key: PYLUCENE-50
> URL: https://issues.apache.org/jira/browse/PYLUCENE-50
> Project: PyLucene
>  Issue Type: Bug
> Environment: Python 3 official docker container, 8.1.1rc
>Reporter: A. Coady
>Priority: Critical
>
> The StoredField constructor is interpreting ints as bytes.  It's only 
> reproducing on the 8 rc.
> {{import lucene}}
>  {{lucene.initVM()}}
>  {{from org.apache.lucene import document}}
>   
>  {{field = document.StoredField('', 0)}}
>  {{print(field.numericValue(), field.binaryValue())}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release PyLucene 8.1.1

[Andi Vajda]

The release vote has now been called off due to PYLUCENE-50.

Andi..

On Mon, 17 Jun 2019, Andi Vajda wrote:



On Mon, 17 Jun 2019, David Allouche wrote:


On 17 Jun 2019, at 20:42, Andi Vajda  wrote:


On Mon, 17 Jun 2019, David Allouche wrote:


Thank you, that was very informative.

+0 for this release, I builds and pass my test suite.

But I was unable to make a complete integration test because I do not 
have a proper index format migration infrastructure and my index is made 
of incompletely-upgraded lucene6 and lucene7 segments.


So far, you're the only one who has cast a vote.
No one else, no PMC member, none of the longtime users, no one.


Marc Jeurissen did +1.


Oops, I missed this message. It is in my inbox, however.
My bad.
Milo just voted +0.

So, after the vote being open for a week, we have:
 - one +1 vote (Marc's)
 - two +0 votes (yours and Milo's)
 - one +1 PMC vote (mine)

For a release to happen, we need two more PMC votes.

Andi..

Message-Id: 
<5cff4939.1c69fb81.2d212.8174smtpin_added_miss...@mx.google.com>


And since you posted your message, Milo H Fields III did +0.

So there _is_ interest.

I do have a long-term need for PyLucene, but I really just need one release 
per major version to have an index upgrade path. But it seems like Lucene 
Core make major releases quite frequently nowadays.

[jira] [Commented] (PYLUCENE-50) StoredField of an int has the wrong type.

[Andi Vajda (JIRA)]

[ 
https://issues.apache.org/jira/browse/PYLUCENE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866171#comment-16866171
 ] 

Andi Vajda commented on PYLUCENE-50:


For the longest time, it was possible to pass an int where a byte was expected. 
This overlap was intentional and, probably, for convenience. The fix to 
PYLUCENE-47 changed the order in which signatures were considered and may have 
caused this long latent bug to now bite.
To fix this, I removed the overlap. To pass a byte, use b'x'.


> StoredField of an int has the wrong type.
> -
>
> Key: PYLUCENE-50
> URL: https://issues.apache.org/jira/browse/PYLUCENE-50
> Project: PyLucene
>  Issue Type: Bug
> Environment: Python 3 official docker container, 8.1.1rc
>Reporter: A. Coady
>Priority: Critical
>
> The StoredField constructor is interpreting ints as bytes.  It's only 
> reproducing on the 8 rc.
> {{import lucene}}
>  {{lucene.initVM()}}
>  {{from org.apache.lucene import document}}
>   
>  {{field = document.StoredField('', 0)}}
>  {{print(field.numericValue(), field.binaryValue())}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (PYLUCENE-50) StoredField of an int has the wrong type.

[Andi Vajda (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PYLUCENE-50?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andi Vajda resolved PYLUCENE-50.

Resolution: Fixed

fixed in rev 1861553

> StoredField of an int has the wrong type.
> -
>
> Key: PYLUCENE-50
> URL: https://issues.apache.org/jira/browse/PYLUCENE-50
> Project: PyLucene
>  Issue Type: Bug
> Environment: Python 3 official docker container, 8.1.1rc
>Reporter: A. Coady
>Priority: Critical
>
> The StoredField constructor is interpreting ints as bytes.  It's only 
> reproducing on the 8 rc.
> {{import lucene}}
>  {{lucene.initVM()}}
>  {{from org.apache.lucene import document}}
>   
>  {{field = document.StoredField('', 0)}}
>  {{print(field.numericValue(), field.binaryValue())}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [jira] [Created] (PYLUCENE-50) StoredField of an int has the wrong type.

[Andi Vajda]

This calls off the vote...

Andi..

On Tue, 18 Jun 2019, A. Coady (JIRA) wrote:


A. Coady created PYLUCENE-50:


Summary: StoredField of an int has the wrong type.
Key: PYLUCENE-50
URL: https://issues.apache.org/jira/browse/PYLUCENE-50
Project: PyLucene
 Issue Type: Bug
Environment: Python 3 official docker container, 8.1.1rc
   Reporter: A. Coady


The StoredField constructor is interpreting ints as bytes.  It's only 
reproducing on the 8 rc.

{{import lucene}}
{{lucene.initVM()}}
{{from org.apache.lucene import document}}


?? {{field = document.StoredField('', 0)}}

{{print(field.numericValue(), field.binaryValue())}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (LUCENE-8866) Remove ICU dependency of kuromoji tools/test-tools

[Robert Muir (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Robert Muir updated LUCENE-8866:

Attachment: LUCENE-8866.patch

> Remove ICU dependency of kuromoji tools/test-tools
> --
>
> Key: LUCENE-8866
> URL: https://issues.apache.org/jira/browse/LUCENE-8866
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-8866.patch
>
>
> The tooling stuff has an off-by-default option to normalize entries, 
> currently using the ICU api.
> But I think since its off-by-default, and just doing NFKC normalization at 
> dictionary-build-time, its a better tradeoff to use the JDK here?
> I would rather remove the ICU dependency for the tooling and look at 
> simplifying the build to have less modules (e.g. investigate moving the 
> tooling and tests into src/java and src/tools, so that [~msoko...@gmail.com] 
> new tests in LUCENE-8863 are running by default, dictionary tool is shipped 
> as a commandline tool in the JAR, etc)
> "ant regenerate" should be enough to prevent any chicken-and-eggs in the 
> dictionary construction code, so I don't think we need separate modules to 
> enforce it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8866) Remove ICU dependency of kuromoji tools/test-tools

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866133#comment-16866133
 ] 

Robert Muir commented on LUCENE-8866:
-

Simple patch, I didn't move any code around, just removed the external dep.

> Remove ICU dependency of kuromoji tools/test-tools
> --
>
> Key: LUCENE-8866
> URL: https://issues.apache.org/jira/browse/LUCENE-8866
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Robert Muir
>Priority: Major
> Attachments: LUCENE-8866.patch
>
>
> The tooling stuff has an off-by-default option to normalize entries, 
> currently using the ICU api.
> But I think since its off-by-default, and just doing NFKC normalization at 
> dictionary-build-time, its a better tradeoff to use the JDK here?
> I would rather remove the ICU dependency for the tooling and look at 
> simplifying the build to have less modules (e.g. investigate moving the 
> tooling and tests into src/java and src/tools, so that [~msoko...@gmail.com] 
> new tests in LUCENE-8863 are running by default, dictionary tool is shipped 
> as a commandline tool in the JAR, etc)
> "ant regenerate" should be enough to prevent any chicken-and-eggs in the 
> dictionary construction code, so I don't think we need separate modules to 
> enforce it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8866) Remove ICU dependency of kuromoji tools/test-tools

[Robert Muir (JIRA)]

Robert Muir created LUCENE-8866:
---

 Summary: Remove ICU dependency of kuromoji tools/test-tools
 Key: LUCENE-8866
 URL: https://issues.apache.org/jira/browse/LUCENE-8866
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Robert Muir


The tooling stuff has an off-by-default option to normalize entries, currently 
using the ICU api.

But I think since its off-by-default, and just doing NFKC normalization at 
dictionary-build-time, its a better tradeoff to use the JDK here?

I would rather remove the ICU dependency for the tooling and look at 
simplifying the build to have less modules (e.g. investigate moving the tooling 
and tests into src/java and src/tools, so that [~msoko...@gmail.com] new tests 
in LUCENE-8863 are running by default, dictionary tool is shipped as a 
commandline tool in the JAR, etc)

"ant regenerate" should be enough to prevent any chicken-and-eggs in the 
dictionary construction code, so I don't think we need separate modules to 
enforce it.




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr Developer content

[Alexandre Rafalovitch]

Does it need to go to Confluence? I know Apache built an export tool,
but is there a way to dump the whole thing into a text archive? Or
both :-)

I am wondering if this could be a good opportunity for dog-fooding.
Load the wiki export into Solr, cross-match against RefGuide, manually
inspect the differences, etc. I already have the code for pulling the
Ref-Guide into Solr split at the lowest-header level. A similar thing
could be done for Wiki and then we do "more like this" or "similarity"
or some such.

Also, how much non-Javadoc information actually exists for Lucene? How
many pages would a Lucene branch have. An honest question, I never
really paid attention to the documentation proportions.

Regards,
   Alex.


On Mon, 17 Jun 2019 at 17:32, David Smiley  wrote:
>
> Great plan, Jan!
>
> A sticky bit of this I think is how to remove old stuff.  It's easy to keep 
> content around forever but it gets stale and clutters things up with better 
> content.  Maybe if I/someone wants to remove content, we send out a proposal 
> to the list with links for easy peer review of what's at stake, and with a 
> justification.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>
>
> On Mon, Jun 17, 2019 at 4:22 PM Joel Bernstein  wrote:
>>
>> +1 for more asciidoc guides. I find these to be extremely useful anytime I 
>> run across these on projects.
>>
>> I'd be happy to add developer level docs in Streaming Expressions / Math 
>> Expressions.
>>
>>
>> Joel Bernstein
>> http://joelsolr.blogspot.com/
>>
>>
>> On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl  wrote:
>>>
>>> Hi devs,
>>>
>>> Today we have mainly two sources of developer documentation (apart from 
>>> Javadoc and refGuide):
>>>
>>> * The websites. Very short instructions and linking to WIKI for in-depth
>>> * The old Moin wikis at wiki.apache.org with more details
>>>
>>> Soon the old Moin wiki is being discontinued and I plan to migrate that 
>>> content to Confluence this week, see 
>>> https://issues.apache.org/jira/browse/LUCENE-8858 and 
>>> https://issues.apache.org/jira/browse/SOLR-13548
>>>
>>> So the first step will be to just start using Confluence instead of Moin. 
>>> Help appreciated with the cleanup once the first migration is done in the 
>>> two JIRAs above. A LOT of the content in old WIKIs is outdated and a big 
>>> cleanup once this is in Confluence is highly needed!
>>>
>>>
>>> Someone has also suggested to move most developer resources found in the 
>>> WIKI into the main GIT code tree, so you have it right there with your git 
>>> clone. What I want to discuss here is more detailed how that would look 
>>> like and what info to move over.
>>>
>>> One idea is to create one or more Asciidoc guides in the source tree, e.g
>>>
>>> * /dev-docs : Common info i.e. Git, Pull requests, building, doing releases 
>>> etc. Publish in TLP site
>>> * /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene 
>>> web site
>>> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web 
>>> site
>>>
>>> These will be built with Jekyll by Jenkins, into nice HTML guides and 
>>> published to the web sites.
>>>
>>> There may be other ways to do this as well, such as creating a new git repo 
>>> for dev docs, but I think people have good experience from Solr's ref-guide 
>>> with keeping code and docs in sync. What do you think?
>>>
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - www.cominvent.com
>>>
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] rmuir commented on a change in pull request #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

rmuir commented on a change in pull request #722: LUCENE-8863: handle some edge 
cases in Kuromoji DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#discussion_r294575405
 
 

 ##
 File path: lucene/analysis/kuromoji/build.xml
 ##
 @@ -136,8 +136,8 @@
  
   
 
-  
-
+  
+
 
 Review comment:
   just an idea for the future: if we are fixing this tooling (e.g. replacing 
bogus asserts with real checks so it is usable like this PR does), maybe we 
should simply move all the `tools*` stuff and tests into the regular source 
tree?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] rmuir commented on issue #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

rmuir commented on issue #722: LUCENE-8863: handle some edge cases in Kuromoji 
DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#issuecomment-502904963
 
 
   +1, thank you for taking the time to do these tests. Can we open a followup 
issue to fix whatever we need to fix so that your tests actually run every time 
with `ant test`?
   
   I think its an important step so that we can support different dictionary 
variants and so on, but it doesn't need to block this PR.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (PYLUCENE-50) StoredField of an int has the wrong type.

[A. Coady (JIRA)]

A. Coady created PYLUCENE-50:


 Summary: StoredField of an int has the wrong type.
 Key: PYLUCENE-50
 URL: https://issues.apache.org/jira/browse/PYLUCENE-50
 Project: PyLucene
  Issue Type: Bug
 Environment: Python 3 official docker container, 8.1.1rc
Reporter: A. Coady


The StoredField constructor is interpreting ints as bytes.  It's only 
reproducing on the 8 rc.

{{import lucene}}
 {{lucene.initVM()}}
 {{from org.apache.lucene import document}}
  
 {{field = document.StoredField('', 0)}}
 {{print(field.numericValue(), field.binaryValue())}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release PyLucene 8.1.1

[Andi Vajda]

On Mon, 17 Jun 2019, David Allouche wrote:


On 17 Jun 2019, at 20:42, Andi Vajda  wrote:


On Mon, 17 Jun 2019, David Allouche wrote:


Thank you, that was very informative.

+0 for this release, I builds and pass my test suite.

But I was unable to make a complete integration test because I do not have a 
proper index format migration infrastructure and my index is made of 
incompletely-upgraded lucene6 and lucene7 segments.


So far, you're the only one who has cast a vote.
No one else, no PMC member, none of the longtime users, no one.


Marc Jeurissen did +1.


Oops, I missed this message. It is in my inbox, however.
My bad.
Milo just voted +0.

So, after the vote being open for a week, we have:
  - one +1 vote (Marc's)
  - two +0 votes (yours and Milo's)
  - one +1 PMC vote (mine)

For a release to happen, we need two more PMC votes.

Andi..


Message-Id: <5cff4939.1c69fb81.2d212.8174smtpin_added_miss...@mx.google.com>

And since you posted your message, Milo H Fields III did +0.

So there _is_ interest.

I do have a long-term need for PyLucene, but I really just need one 
release per major version to have an index upgrade path. But it seems like 
Lucene Core make major releases quite frequently nowadays.

[GitHub] [lucene-solr] uschindler commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

uschindler commented on a change in pull request #725: LUCENE-8865: Use 
incoming thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294557617
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -639,13 +639,15 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length - 1) { // execute the last on the caller 
thread
 
 Review comment:
   Yeah, would be better!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8865) Use incoming thread for execution if IndexSearcher has an executor

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866073#comment-16866073
 ] 

Uwe Schindler commented on LUCENE-8865:
---

+1

>  Use incoming thread for execution if IndexSearcher has an executor
> ---
>
> Key: LUCENE-8865
> URL: https://issues.apache.org/jira/browse/LUCENE-8865
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Today we don't utilize the incoming thread for a search when IndexSearcher
> has an executor. This thread is only idleing but can be used to execute a 
> search
> once all other collectors are dispatched.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-8.x - Build # 244 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/244/

3 tests failed.
FAILED:  
org.apache.solr.cloud.TestWaitForStateWithJettyShutdowns.testWaitForStateAfterShutDown

Error Message:
IOException occurred when talking to server at: http://127.0.0.1:32780/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occurred when 
talking to server at: http://127.0.0.1:32780/solr
at 
__randomizedtesting.SeedInfo.seed([85108468646543C0:639B3ED781171FDB]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:670)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1128)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:897)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:829)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:228)
at 
org.apache.solr.cloud.TestWaitForStateWithJettyShutdowns.testWaitForStateAfterShutDown(TestWaitForStateWithJettyShutdowns.java:53)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)

[jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866042#comment-16866042
 ] 

ASF subversion and git services commented on SOLR-13490:


Commit 7eb8703df64b4fdda8113ddcbcd0b4d2413ecc38 in lucene-solr's branch 
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7eb8703 ]

SOLR-13490: fix TestWaitForStateWithJettyShutdowns to use correct (randomized) 
JettyConfig


> waitForState/registerCollectionStateWatcher can see stale liveNodes data due 
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Fix For: master (9.0), 8.2
>
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, 
> SOLR-13490.patch
>
>
> I was investigating some failures in 
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the 
> hunch that {{waitForState}} wasn't ensuring that the predicates registered 
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way 
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in 
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only 
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to 
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / 
> {{CollectionStatePredicate}} are called, they get the "fresh" 
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only 
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - 
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas 
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means 
> there is a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if 
> "replicaX" of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new 
> DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and 
> doesn't take any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified 
> fired with the updated liveNodes (unless/until another {{state.json}} change 
> is made for collectionA.
> 
> While this is definitely be problematic in _tests_ that deal with node 
> lifecyle and use things like {{SolrCloudTestCase.waitForState(..., 
> SolrCloudTestCase.clusterShape(...))}} to check for the expected 
> shards/replicas, a cursory search of how/where 
> {{ZkStateReader.waitForState(...)}} and 
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core 
> suggests that could also lead to bad behavior in situations like reacting to 
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for 
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by 
> the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866041#comment-16866041
 ] 

ASF subversion and git services commented on SOLR-13490:


Commit 592d10d7ce82c9d0f38ea66f3c2f1a4113649728 in lucene-solr's branch 
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=592d10d ]

SOLR-13490: fix TestWaitForStateWithJettyShutdowns to use correct (randomized) 
JettyConfig

(cherry picked from commit 7eb8703df64b4fdda8113ddcbcd0b4d2413ecc38)


> waitForState/registerCollectionStateWatcher can see stale liveNodes data due 
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Fix For: master (9.0), 8.2
>
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, 
> SOLR-13490.patch
>
>
> I was investigating some failures in 
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the 
> hunch that {{waitForState}} wasn't ensuring that the predicates registered 
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way 
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in 
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only 
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to 
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / 
> {{CollectionStatePredicate}} are called, they get the "fresh" 
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only 
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - 
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas 
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means 
> there is a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if 
> "replicaX" of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new 
> DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and 
> doesn't take any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified 
> fired with the updated liveNodes (unless/until another {{state.json}} change 
> is made for collectionA.
> 
> While this is definitely be problematic in _tests_ that deal with node 
> lifecyle and use things like {{SolrCloudTestCase.waitForState(..., 
> SolrCloudTestCase.clusterShape(...))}} to check for the expected 
> shards/replicas, a cursory search of how/where 
> {{ZkStateReader.waitForState(...)}} and 
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core 
> suggests that could also lead to bad behavior in situations like reacting to 
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for 
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by 
> the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866028#comment-16866028
 ] 

Lucene/Solr QA commented on SOLR-7530:
--

| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
19s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  2m  6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  1m 59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  1m 59s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 50m  
9s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 58m 14s{color} | 
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-7530 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12972004/SOLR-7530.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 5a97486 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/441/testReport/ |
| modules | C: solr solr/core solr/test-framework U: solr |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/441/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] [lucene-solr] mikemccand commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

mikemccand commented on a change in pull request #725: LUCENE-8865: Use 
incoming thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294538579
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -639,13 +639,15 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length - 1) { // execute the last on the caller 
thread
 
 Review comment:
   Can we change `for` loop to only iterate up to `leafSlices.length - 1` and 
then do this afterwards?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated SOLR-7530:
---
Attachment: SOLR-7530.patch

> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] dsmiley commented on issue #726: LUCENE-8632: New XYShape Field and Queries for indexing and searching general cartesian geometries

[GitBox]

dsmiley commented on issue #726: LUCENE-8632: New XYShape Field and Queries for 
indexing and searching general cartesian geometries
URL: https://github.com/apache/lucene-solr/pull/726#issuecomment-502861247
 
 
   This all sounds great; I like the use of the "XY" as a naming prefix to help 
clarify the coordinate system so we don't confuse it with LatLon.
   
   In your mind... with the increase spatial code in core, are you still 
resolute in your opinion of Lucene-core having all this spatial code?  I 
mean... wow... it's not like Lucene core doesn't have a rich module system 
already.  Yeah I know this is kinda settled already but this just gnaws at me.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr Developer content

[Adrien Grand]

+1

On Mon, Jun 17, 2019 at 10:18 PM Jan Høydahl  wrote:
>
> Hi devs,
>
> Today we have mainly two sources of developer documentation (apart from 
> Javadoc and refGuide):
>
> * The websites. Very short instructions and linking to WIKI for in-depth
> * The old Moin wikis at wiki.apache.org with more details
>
> Soon the old Moin wiki is being discontinued and I plan to migrate that 
> content to Confluence this week, see 
> https://issues.apache.org/jira/browse/LUCENE-8858 and 
> https://issues.apache.org/jira/browse/SOLR-13548
>
> So the first step will be to just start using Confluence instead of Moin. 
> Help appreciated with the cleanup once the first migration is done in the two 
> JIRAs above. A LOT of the content in old WIKIs is outdated and a big cleanup 
> once this is in Confluence is highly needed!
>
>
> Someone has also suggested to move most developer resources found in the WIKI 
> into the main GIT code tree, so you have it right there with your git clone. 
> What I want to discuss here is more detailed how that would look like and 
> what info to move over.
>
> One idea is to create one or more Asciidoc guides in the source tree, e.g
>
> * /dev-docs : Common info i.e. Git, Pull requests, building, doing releases 
> etc. Publish in TLP site
> * /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene 
> web site
> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web site
>
> These will be built with Jekyll by Jenkins, into nice HTML guides and 
> published to the web sites.
>
> There may be other ways to do this as well, such as creating a new git repo 
> for dev docs, but I think people have good experience from Solr's ref-guide 
> with keeping code and docs in sync. What do you think?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>


-- 
Adrien

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr Developer content

[David Smiley]

Great plan, Jan!

A sticky bit of this I think is how to remove old stuff.  It's easy to keep
content around forever but it gets stale and clutters things up with better
content.  Maybe if I/someone wants to remove content, we send out a
proposal to the list with links for easy peer review of what's at stake,
and with a justification.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Mon, Jun 17, 2019 at 4:22 PM Joel Bernstein  wrote:

> +1 for more asciidoc guides. I find these to be extremely useful anytime I
> run across these on projects.
>
> I'd be happy to add developer level docs in Streaming Expressions / Math
> Expressions.
>
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
>
> On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl  wrote:
>
>> Hi devs,
>>
>> Today we have mainly two sources of developer documentation (apart from
>> Javadoc and refGuide):
>>
>> * The websites. Very short instructions and linking to WIKI for in-depth
>> * The old Moin wikis at wiki.apache.org with more details
>>
>> Soon the old Moin wiki is being discontinued and I plan to migrate that
>> content to Confluence this week, see
>> https://issues.apache.org/jira/browse/LUCENE-8858 and
>> https://issues.apache.org/jira/browse/SOLR-13548
>>
>> So the first step will be to just start using Confluence instead of Moin.
>> Help appreciated with the cleanup once the first migration is done in the
>> two JIRAs above. A LOT of the content in old WIKIs is outdated and a big
>> cleanup once this is in Confluence is highly needed!
>>
>>
>> Someone has also suggested to move most developer resources found in the
>> WIKI into the main GIT code tree, so you have it right there with your git
>> clone. What I want to discuss here is more detailed how that would look
>> like and what info to move over.
>>
>> One idea is to create one or more Asciidoc guides in the source tree, e.g
>>
>> * /dev-docs : Common info i.e. Git, Pull requests, building, doing
>> releases etc. Publish in TLP site
>> * /lucene/dev-guide : Lucene-specific developer content. Publish in
>> Lucene web site
>> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web
>> site
>>
>> These will be built with Jekyll by Jenkins, into nice HTML guides and
>> published to the web sites.
>>
>> There may be other ways to do this as well, such as creating a new git
>> repo for dev docs, but I think people have good experience from Solr's
>> ref-guide with keeping code and docs in sync. What do you think?
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

[GitHub] [lucene-solr] msokolov commented on issue #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

msokolov commented on issue #722: LUCENE-8863: handle some edge cases in 
Kuromoji DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#issuecomment-502856635
 
 
   I posted an updated patch that addresses the comments, and also adds a test 
that exercises the builder with some test data. In order to test, I needed to 
add the ability to load an "external" dictionary; this enables loading a system 
dictionary from an alternate location on the classpath, or from an external 
file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10291) Add match Boolean Stream Evaluator to support regex matching

[Joel Bernstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865983#comment-16865983
 ] 

Joel Bernstein commented on SOLR-10291:
---

Time to get this powerful capability into Streaming Expressions.

> Add match Boolean Stream Evaluator to support regex matching
> 
>
> Key: SOLR-10291
> URL: https://issues.apache.org/jira/browse/SOLR-10291
> Project: Solr
>  Issue Type: New Feature
>  Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
>
> It would be good to be able to match any regex as part of a having expression 
> etc...
> Lucene/Solr does support regex queries but the match Evaluator would match 
> against un-analyzed blocks of texts, so it would eliminate any confusion 
> about what terms it was actually matching against.
> This would support use cases like alerting on sensitive content that appear 
> in documents such as SSN and credit card numbers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-10291) Add match Boolean Stream Evaluator to support regex matching

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-10291:
--
Component/s: streaming expressions

> Add match Boolean Stream Evaluator to support regex matching
> 
>
> Key: SOLR-10291
> URL: https://issues.apache.org/jira/browse/SOLR-10291
> Project: Solr
>  Issue Type: New Feature
>  Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
>
> It would be good to be able to match any regex as part of a having expression 
> etc...
> Lucene/Solr does support regex queries but the match Evaluator would match 
> against un-analyzed blocks of texts, so it would eliminate any confusion 
> about what terms it was actually matching against.
> This would support use cases like alerting on sensitive content that appear 
> in documents such as SSN and credit card numbers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-10291) Add match Boolean Stream Evaluator to support regex matching

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-10291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein reassigned SOLR-10291:
-

Assignee: Joel Bernstein

> Add match Boolean Stream Evaluator to support regex matching
> 
>
> Key: SOLR-10291
> URL: https://issues.apache.org/jira/browse/SOLR-10291
> Project: Solr
>  Issue Type: New Feature
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
>
> It would be good to be able to match any regex as part of a having expression 
> etc...
> Lucene/Solr does support regex queries but the match Evaluator would match 
> against un-analyzed blocks of texts, so it would eliminate any confusion 
> about what terms it was actually matching against.
> This would support use cases like alerting on sensitive content that appear 
> in documents such as SSN and credit card numbers. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk1.8.0_201) - Build # 722 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/722/
Java: 64bit/jdk1.8.0_201 -XX:-UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  org.apache.solr.cloud.ShardRoutingTest.test

Error Message:
Error from server at https://127.0.0.1:43503/x_qne/if/collection1: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: 
https://127.0.0.1:45187/x_qne/if/collection1_shard1_replica_n4/get

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at https://127.0.0.1:43503/x_qne/if/collection1: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: 
https://127.0.0.1:45187/x_qne/if/collection1_shard1_replica_n4/get
at 
__randomizedtesting.SeedInfo.seed([B8BEB5EB6E841928:30EA8A31C07874D0]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:656)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:987)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1002)
at 
org.apache.solr.cloud.ShardRoutingTest.doAtomicUpdate(ShardRoutingTest.java:307)
at 
org.apache.solr.cloud.ShardRoutingTest.test(ShardRoutingTest.java:111)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[jira] [Created] (SOLR-13556) Allow length Stream evaluator to return the length of a string

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-13556:
-

 Summary: Allow length Stream evaluator to return the length of a 
string
 Key: SOLR-13556
 URL: https://issues.apache.org/jira/browse/SOLR-13556
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein


Currently the *length* Stream Evaluator returns the length of an array. This 
ticket will allow the length Stream Evaluator to also return the length of a 
string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13556) Allow the length Stream Evaluator to return the length of a string

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-13556:
--
Summary: Allow the length Stream Evaluator to return the length of a string 
 (was: Allow length Stream evaluator to return the length of a string)

> Allow the length Stream Evaluator to return the length of a string
> --
>
> Key: SOLR-13556
> URL: https://issues.apache.org/jira/browse/SOLR-13556
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Priority: Major
>
> Currently the *length* Stream Evaluator returns the length of an array. This 
> ticket will allow the length Stream Evaluator to also return the length of a 
> string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13554) Add robust flag to movingMAD Stream Evaluation

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-13554:
--
Summary: Add robust flag to movingMAD Stream Evaluation  (was: Add robust 
flag to movingMAD Streaming Evaluation)

> Add robust flag to movingMAD Stream Evaluation
> --
>
> Key: SOLR-13554
> URL: https://issues.apache.org/jira/browse/SOLR-13554
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Priority: Major
>
> Currently the movingMAD Stream Evaluator computes a moving *mean* absolute 
> deviation for a time series. This ticket will add a named parameter called 
> "robust" which will calculate a moving *median* absolute deviation. This will 
> be useful for time series anomaly detection use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13555) Add centeredMovingAvg Stream Evaluator

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-13555:
-

 Summary: Add centeredMovingAvg Stream Evaluator
 Key: SOLR-13555
 URL: https://issues.apache.org/jira/browse/SOLR-13555
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein


Currently the movingAvg function calculates a trailing moving average. This 
ticket will add the centeredMovingAvg function which will calculate a centered 
moving avg.

Under the covers the centeredMovingAverage function will use knnRegression to 
calculate the centered moving average.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-13555) Add centeredMovingAvg Stream Evaluator

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein reassigned SOLR-13555:
-

Assignee: Joel Bernstein

> Add centeredMovingAvg Stream Evaluator
> --
>
> Key: SOLR-13555
> URL: https://issues.apache.org/jira/browse/SOLR-13555
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
>
> Currently the movingAvg function calculates a trailing moving average. This 
> ticket will add the centeredMovingAvg function which will calculate a 
> centered moving avg.
> Under the covers the centeredMovingAverage function will use knnRegression to 
> calculate the centered moving average.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13554) Add robust flag to movingMAD Streaming Evaluation

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-13554:
-

 Summary: Add robust flag to movingMAD Streaming Evaluation
 Key: SOLR-13554
 URL: https://issues.apache.org/jira/browse/SOLR-13554
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein


Currently the movingMAD Stream Evaluator computes a moving *mean* absolute 
deviation for a time series. This ticket will add a named parameter called 
"robust" which will calculate a moving *median* absolute deviation. This will 
be useful for time series anomaly detection use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release PyLucene 8.1.1

[David Allouche]

> On 17 Jun 2019, at 20:42, Andi Vajda  wrote:
> 
> 
> On Mon, 17 Jun 2019, David Allouche wrote:
> 
>> Thank you, that was very informative.
>> 
>> +0 for this release, I builds and pass my test suite.
>> 
>> But I was unable to make a complete integration test because I do not have a 
>> proper index format migration infrastructure and my index is made of 
>> incompletely-upgraded lucene6 and lucene7 segments.
> 
> So far, you're the only one who has cast a vote.
> No one else, no PMC member, none of the longtime users, no one.

Marc Jeurissen did +1.
Message-Id: <5cff4939.1c69fb81.2d212.8174smtpin_added_miss...@mx.google.com>

And since you posted your message, Milo H Fields III did +0.

So there _is_ interest.

I do have a long-term need for PyLucene, but I really just need one release per 
major version to have an index upgrade path. But it seems like Lucene Core make 
major releases quite frequently nowadays.

[jira] [Commented] (SOLR-11162) ExternalFileField/FileFloatSource should support a Points based keyField

[Gregg Donovan (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865964#comment-16865964
 ] 

Gregg Donovan commented on SOLR-11162:
--

[~steve_rowe] [~hossman]

What do you have in mind for this? If Points fields don't give a TermsEnum, how 
can we iterate their values? It looks like we'll need to get a PointValues and 
iterate via some sort of IntersectVisitor? Do you think this will be as fast as 
the TermsEnum method? I don't see a way to save the iteration state the way 
TermsEnum does, so I worry about about performance.

> ExternalFileField/FileFloatSource should support a Points based keyField
> 
>
> Key: SOLR-11162
> URL: https://issues.apache.org/jira/browse/SOLR-11162
> Project: Solr
>  Issue Type: Improvement
>Reporter: Steve Rowe
>Priority: Major
>  Labels: numeric-tries-to-points
>
> ExternalFileField/FileFloatSource does not support Points based keyField 
> specification, but it should.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr Developer content

[Joel Bernstein]

+1 for more asciidoc guides. I find these to be extremely useful anytime I
run across these on projects.

I'd be happy to add developer level docs in Streaming Expressions / Math
Expressions.


Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Jun 17, 2019 at 4:18 PM Jan Høydahl  wrote:

> Hi devs,
>
> Today we have mainly two sources of developer documentation (apart from
> Javadoc and refGuide):
>
> * The websites. Very short instructions and linking to WIKI for in-depth
> * The old Moin wikis at wiki.apache.org with more details
>
> Soon the old Moin wiki is being discontinued and I plan to migrate that
> content to Confluence this week, see
> https://issues.apache.org/jira/browse/LUCENE-8858 and
> https://issues.apache.org/jira/browse/SOLR-13548
>
> So the first step will be to just start using Confluence instead of Moin.
> Help appreciated with the cleanup once the first migration is done in the
> two JIRAs above. A LOT of the content in old WIKIs is outdated and a big
> cleanup once this is in Confluence is highly needed!
>
>
> Someone has also suggested to move most developer resources found in the
> WIKI into the main GIT code tree, so you have it right there with your git
> clone. What I want to discuss here is more detailed how that would look
> like and what info to move over.
>
> One idea is to create one or more Asciidoc guides in the source tree, e.g
>
> * /dev-docs : Common info i.e. Git, Pull requests, building, doing
> releases etc. Publish in TLP site
> * /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene
> web site
> * /solr/dev-guide : Solr-specific developer content. Publish in Solr web
> site
>
> These will be built with Jekyll by Jenkins, into nice HTML guides and
> published to the web sites.
>
> There may be other ways to do this as well, such as creating a new git
> repo for dev docs, but I think people have good experience from Solr's
> ref-guide with keeping code and docs in sync. What do you think?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

RE: [VOTE] Release PyLucene 8.1.1

[Milo H Fields III]

+0 (user)


> -Original Message-
> From: Andi Vajda 
> Sent: Monday, June 17, 2019 14:43
> To: pylucene-dev@lucene.apache.org
> Subject: Re: [VOTE] Release PyLucene 8.1.1
> 
> 
> On Mon, 17 Jun 2019, David Allouche wrote:
> 
> > Thank you, that was very informative.
> >
> > +0 for this release, I builds and pass my test suite.
> >
> > But I was unable to make a complete integration test because I do not
> > have a proper index format migration infrastructure and my index is
> > made of incompletely-upgraded lucene6 and lucene7 segments.
> 
> So far, you're the only one who has cast a vote.
> No one else, no PMC member, none of the longtime users, no one.
> 
> Besides the Apache procedural aspect, voting is also a gauge of interest
in a
> project. If there is no user interest in the PyLucene project anymore
maybe
> it's time to stop making releases for a while ?
> 
> PyLucene is a bit special in that it doesn't involve many people for
> development since it is machine-generated by JCC and JCC has been stable
> for a while. In that sense, it is important that actual users of PyLucene
make
> themselves known by voting, not much else goes on on the project's mailing
> list or in the source code.
> 
> If there are actual users showing interest by voting here, I feel more
> confortable then in nagging people on the Lucene PMC for their procedural
> vote.
> 
> Andi..
> 
> >
> > I will post my questions about index upgrading on
> > java-u...@lucene.apache.org .
> 
> 
> >
> > Regards.
> >
> >> On 11 Jun 2019, at 16:50, Andi Vajda  wrote:
> >>
> >>
> >>> On Jun 11, 2019, at 06:30, David Allouche  wrote:
> >>>
> >>> This is maybe a silly question, but what is the purpose of this voting
> process?
> >>
> >> By the rules of Apache, three PMC binding votes are needed to make a
> release. In addition, it's a gauge of general interest in the project.
> >>
> >>> Is this something required by the project governance?
> >>
> >> https://www.apache.org/foundation/voting.html
> >> (see "votes on package releases", in particular)
> >>
> >>> What is the meaning of a vote? Does that mean "I am interested", or
> does it mean "I have tested the latest trunk and it looks good", or
something
> else?
> >>
> >> If you find a bug in the release artifacts (not in the latest trunk)
before the
> release is made, the release is likely to be pulled.
> >>
> >>> What is the typical expected delay for reply? For example, I reserve
> Fridays for technical debt management (including upgrading dependencies),
> so I cannot typically validate a new PyLucene version in less than a week.
> >>
> >> A vote must run for at least 72 hours.
> >> Because you are not on the PMC, your vote falls into the "interest
> gauging" category, is not binding and is considered "best effort".
> >>
> >>> This is probably all common questions with well documented answers. If
> that's the case, then it would be nice to have a link to the answers in
VOTE
> requests.
> >>
> >> https://www.apache.org/foundation/voting.html
> >>
> >> Andi..
> >>
> >>>
>  On 11 Jun 2019, at 00:39, Andi Vajda  wrote:
> 
> 
>  The PyLucene 8.1.1 (rc1) release tracking the recent release of
>  Apache Lucene 8.1.1 is ready.
> 
>  A release candidate is available from:
>  https://dist.apache.org/repos/dist/dev/lucene/pylucene/8.1.1-rc1/
> 
>  PyLucene 8.1.1 is built with JCC 3.5, included in these release
artifacts.
> 
>  JCC 3.5 supports Python 3.3+ (in addition to Python 2.3+).
>  PyLucene may be built with Python 2 or Python 3.
> 
>  Please vote to release these artifacts as PyLucene 8.1.1.
>  Anyone interested in this release can and should vote !
> 
>  Thanks !
> 
>  Andi..
> 
>  ps: the KEYS file for PyLucene release signing is at:
>  https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
>  https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS
> 
>  pps: here is my +1
> >>>
> >
> >


smime.p7s
Description: S/MIME cryptographic signature

Re: [JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still Failing

[Joel Bernstein]

Ok, thanks for the explanation.

Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Jun 17, 2019 at 4:04 PM Chris Hostetter 
wrote:

>
> It is sporadic & seemingly unpredictible & has been happening for
> various people on various machines / OSes since the upgrade to java11 on
> master -- see thread "precommit failures" ...
>
>
> http://mail-archives.apache.org/mod_mbox/lucene-dev/201906.mbox/%3Calpine.DEB.2.11.1906131520060.17915%40tray%3E
>
>
>
> : Date: Mon, 17 Jun 2019 15:57:30 -0400
> : From: Joel Bernstein 
> : Reply-To: dev@lucene.apache.org
> : To: lucene dev 
> : Subject: Re: [JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still
> Failing
> :
> : I saw these failures locally when running precommit, After running clean
> : and clean-jars the precommit passed. Is there an issue with the jars on
> : jenkins?
> :
> :
> :
> : Joel Bernstein
> : http://joelsolr.blogspot.com/
> :
> :
> : On Mon, Jun 17, 2019 at 3:35 PM Apache Jenkins Server <
> : jenk...@builds.apache.org> wrote:
> :
> : > Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3385/
> : >
> : > All tests passed
> : >
> : > Build Log:
> : > [...truncated 65007 lines...]
> : > -ecj-javadoc-lint-src:
> : > [mkdir] Created dir: /tmp/ecj671470942
> : >  [ecj-lint] Compiling 69 source files to /tmp/ecj671470942
> : >  [ecj-lint] invalid Class-Path header in manifest of jar file:
> : >
> /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
> : >  [ecj-lint] invalid Class-Path header in manifest of jar file:
> : >
> /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
> : >  [ecj-lint] --
> : >  [ecj-lint] 1. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 28)
> : >  [ecj-lint] import javax.naming.InitialContext;
> : >  [ecj-lint]^^^
> : >  [ecj-lint] The type javax.naming.InitialContext is not accessible
> : >  [ecj-lint] --
> : >  [ecj-lint] 2. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 29)
> : >  [ecj-lint] import javax.naming.NamingException;
> : >  [ecj-lint]
> : >  [ecj-lint] The type javax.naming.NamingException is not accessible
> : >  [ecj-lint] --
> : >  [ecj-lint] 3. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 182)
> : >  [ecj-lint] c = getFromJndi(initProps, jndiName);
> : >  [ecj-lint] ^^^
> : >  [ecj-lint] The method getFromJndi(Properties, String) from the type
> new
> : > Callable(){} refers to the missing type NamingException
> : >  [ecj-lint] --
> : >  [ecj-lint] 4. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 245)
> : >  [ecj-lint] private Connection getFromJndi(final Properties
> initProps,
> : > final String jndiName) throws NamingException,
> : >  [ecj-lint]
> : >   ^^^
> : >  [ecj-lint] NamingException cannot be resolved to a type
> : >  [ecj-lint] --
> : >  [ecj-lint] 5. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 249)
> : >  [ecj-lint] InitialContext ctx =  new InitialContext();
> : >  [ecj-lint] ^^
> : >  [ecj-lint] InitialContext cannot be resolved to a type
> : >  [ecj-lint] --
> : >  [ecj-lint] 6. ERROR in
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> : > (at line 249)
> : >  [ecj-lint] InitialContext ctx =  new InitialContext();
> : >  [ecj-lint]   ^^
> : >  [ecj-lint] InitialContext cannot be resolved to a type
> : >  [ecj-lint] --
> : >  [ecj-lint] 6 problems (6 errors)
> : >
> : > BUILD FAILED
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:634:
> : > The following error occurred while executing this line:
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101:
> : > The following error occurred while executing this line:
> : >
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:651:
> : > The following error occurred while executing this line:
> : >
> 

Lucene/Solr Developer content

[Jan Høydahl]

Hi devs,

Today we have mainly two sources of developer documentation (apart from Javadoc 
and refGuide):

* The websites. Very short instructions and linking to WIKI for in-depth
* The old Moin wikis at wiki.apache.org with more details

Soon the old Moin wiki is being discontinued and I plan to migrate that content 
to Confluence this week, see https://issues.apache.org/jira/browse/LUCENE-8858 
and https://issues.apache.org/jira/browse/SOLR-13548

So the first step will be to just start using Confluence instead of Moin. Help 
appreciated with the cleanup once the first migration is done in the two JIRAs 
above. A LOT of the content in old WIKIs is outdated and a big cleanup once 
this is in Confluence is highly needed!


Someone has also suggested to move most developer resources found in the WIKI 
into the main GIT code tree, so you have it right there with your git clone. 
What I want to discuss here is more detailed how that would look like and what 
info to move over.

One idea is to create one or more Asciidoc guides in the source tree, e.g

* /dev-docs : Common info i.e. Git, Pull requests, building, doing releases 
etc. Publish in TLP site
* /lucene/dev-guide : Lucene-specific developer content. Publish in Lucene web 
site
* /solr/dev-guide : Solr-specific developer content. Publish in Solr web site

These will be built with Jekyll by Jenkins, into nice HTML guides and published 
to the web sites.

There may be other ways to do this as well, such as creating a new git repo for 
dev docs, but I think people have good experience from Solr's ref-guide with 
keeping code and docs in sync. What do you think?

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still Failing

[Chris Hostetter]

It is sporadic & seemingly unpredictible & has been happening for 
various people on various machines / OSes since the upgrade to java11 on 
master -- see thread "precommit failures" ...

http://mail-archives.apache.org/mod_mbox/lucene-dev/201906.mbox/%3Calpine.DEB.2.11.1906131520060.17915%40tray%3E



: Date: Mon, 17 Jun 2019 15:57:30 -0400
: From: Joel Bernstein 
: Reply-To: dev@lucene.apache.org
: To: lucene dev 
: Subject: Re: [JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still Failing
: 
: I saw these failures locally when running precommit, After running clean
: and clean-jars the precommit passed. Is there an issue with the jars on
: jenkins?
: 
: 
: 
: Joel Bernstein
: http://joelsolr.blogspot.com/
: 
: 
: On Mon, Jun 17, 2019 at 3:35 PM Apache Jenkins Server <
: jenk...@builds.apache.org> wrote:
: 
: > Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3385/
: >
: > All tests passed
: >
: > Build Log:
: > [...truncated 65007 lines...]
: > -ecj-javadoc-lint-src:
: > [mkdir] Created dir: /tmp/ecj671470942
: >  [ecj-lint] Compiling 69 source files to /tmp/ecj671470942
: >  [ecj-lint] invalid Class-Path header in manifest of jar file:
: > 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
: >  [ecj-lint] invalid Class-Path header in manifest of jar file:
: > 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
: >  [ecj-lint] --
: >  [ecj-lint] 1. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 28)
: >  [ecj-lint] import javax.naming.InitialContext;
: >  [ecj-lint]^^^
: >  [ecj-lint] The type javax.naming.InitialContext is not accessible
: >  [ecj-lint] --
: >  [ecj-lint] 2. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 29)
: >  [ecj-lint] import javax.naming.NamingException;
: >  [ecj-lint]
: >  [ecj-lint] The type javax.naming.NamingException is not accessible
: >  [ecj-lint] --
: >  [ecj-lint] 3. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 182)
: >  [ecj-lint] c = getFromJndi(initProps, jndiName);
: >  [ecj-lint] ^^^
: >  [ecj-lint] The method getFromJndi(Properties, String) from the type new
: > Callable(){} refers to the missing type NamingException
: >  [ecj-lint] --
: >  [ecj-lint] 4. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 245)
: >  [ecj-lint] private Connection getFromJndi(final Properties initProps,
: > final String jndiName) throws NamingException,
: >  [ecj-lint]
: >   ^^^
: >  [ecj-lint] NamingException cannot be resolved to a type
: >  [ecj-lint] --
: >  [ecj-lint] 5. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 249)
: >  [ecj-lint] InitialContext ctx =  new InitialContext();
: >  [ecj-lint] ^^
: >  [ecj-lint] InitialContext cannot be resolved to a type
: >  [ecj-lint] --
: >  [ecj-lint] 6. ERROR in
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
: > (at line 249)
: >  [ecj-lint] InitialContext ctx =  new InitialContext();
: >  [ecj-lint]   ^^
: >  [ecj-lint] InitialContext cannot be resolved to a type
: >  [ecj-lint] --
: >  [ecj-lint] 6 problems (6 errors)
: >
: > BUILD FAILED
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:634:
: > The following error occurred while executing this line:
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101:
: > The following error occurred while executing this line:
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:651:
: > The following error occurred while executing this line:
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/common-build.xml:479:
: > The following error occurred while executing this line:
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2009:
: > The following error occurred while executing this line:
: > 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2048:
: > 

Re: [JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still Failing

[Joel Bernstein]

I saw these failures locally when running precommit, After running clean
and clean-jars the precommit passed. Is there an issue with the jars on
jenkins?



Joel Bernstein
http://joelsolr.blogspot.com/


On Mon, Jun 17, 2019 at 3:35 PM Apache Jenkins Server <
jenk...@builds.apache.org> wrote:

> Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3385/
>
> All tests passed
>
> Build Log:
> [...truncated 65007 lines...]
> -ecj-javadoc-lint-src:
> [mkdir] Created dir: /tmp/ecj671470942
>  [ecj-lint] Compiling 69 source files to /tmp/ecj671470942
>  [ecj-lint] invalid Class-Path header in manifest of jar file:
> /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
>  [ecj-lint] invalid Class-Path header in manifest of jar file:
> /home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
>  [ecj-lint] --
>  [ecj-lint] 1. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 28)
>  [ecj-lint] import javax.naming.InitialContext;
>  [ecj-lint]^^^
>  [ecj-lint] The type javax.naming.InitialContext is not accessible
>  [ecj-lint] --
>  [ecj-lint] 2. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 29)
>  [ecj-lint] import javax.naming.NamingException;
>  [ecj-lint]
>  [ecj-lint] The type javax.naming.NamingException is not accessible
>  [ecj-lint] --
>  [ecj-lint] 3. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 182)
>  [ecj-lint] c = getFromJndi(initProps, jndiName);
>  [ecj-lint] ^^^
>  [ecj-lint] The method getFromJndi(Properties, String) from the type new
> Callable(){} refers to the missing type NamingException
>  [ecj-lint] --
>  [ecj-lint] 4. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 245)
>  [ecj-lint] private Connection getFromJndi(final Properties initProps,
> final String jndiName) throws NamingException,
>  [ecj-lint]
>   ^^^
>  [ecj-lint] NamingException cannot be resolved to a type
>  [ecj-lint] --
>  [ecj-lint] 5. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 249)
>  [ecj-lint] InitialContext ctx =  new InitialContext();
>  [ecj-lint] ^^
>  [ecj-lint] InitialContext cannot be resolved to a type
>  [ecj-lint] --
>  [ecj-lint] 6. ERROR in
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
> (at line 249)
>  [ecj-lint] InitialContext ctx =  new InitialContext();
>  [ecj-lint]   ^^
>  [ecj-lint] InitialContext cannot be resolved to a type
>  [ecj-lint] --
>  [ecj-lint] 6 problems (6 errors)
>
> BUILD FAILED
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:634:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:651:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/common-build.xml:479:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2009:
> The following error occurred while executing this line:
> /home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2048:
> Compile failed; see the compiler error output for details.
>
> Total time: 238 minutes 54 seconds
> Build step 'Invoke Ant' marked build as failure
> Archiving artifacts
> Recording test results
> Email was triggered for: Failure - Any
> Sending email for trigger: Failure - Any
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-13490:

   Resolution: Fixed
Fix Version/s: 8.2
   master (9.0)
   Status: Resolved  (was: Patch Available)

> waitForState/registerCollectionStateWatcher can see stale liveNodes data due 
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Fix For: master (9.0), 8.2
>
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, 
> SOLR-13490.patch
>
>
> I was investigating some failures in 
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the 
> hunch that {{waitForState}} wasn't ensuring that the predicates registered 
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way 
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in 
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only 
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to 
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / 
> {{CollectionStatePredicate}} are called, they get the "fresh" 
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only 
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - 
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas 
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means 
> there is a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if 
> "replicaX" of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new 
> DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and 
> doesn't take any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified 
> fired with the updated liveNodes (unless/until another {{state.json}} change 
> is made for collectionA.
> 
> While this is definitely be problematic in _tests_ that deal with node 
> lifecyle and use things like {{SolrCloudTestCase.waitForState(..., 
> SolrCloudTestCase.clusterShape(...))}} to check for the expected 
> shards/replicas, a cursory search of how/where 
> {{ZkStateReader.waitForState(...)}} and 
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core 
> suggests that could also lead to bad behavior in situations like reacting to 
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for 
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by 
> the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-master - Build # 3385 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-master/3385/

All tests passed

Build Log:
[...truncated 65007 lines...]
-ecj-javadoc-lint-src:
[mkdir] Created dir: /tmp/ecj671470942
 [ecj-lint] Compiling 69 source files to /tmp/ecj671470942
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet/jars/org.restlet-2.3.0.jar
 [ecj-lint] invalid Class-Path header in manifest of jar file: 
/home/jenkins/.ivy2/cache/org.restlet.jee/org.restlet.ext.servlet/jars/org.restlet.ext.servlet-2.3.0.jar
 [ecj-lint] --
 [ecj-lint] 1. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 28)
 [ecj-lint] import javax.naming.InitialContext;
 [ecj-lint]^^^
 [ecj-lint] The type javax.naming.InitialContext is not accessible
 [ecj-lint] --
 [ecj-lint] 2. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 29)
 [ecj-lint] import javax.naming.NamingException;
 [ecj-lint]
 [ecj-lint] The type javax.naming.NamingException is not accessible
 [ecj-lint] --
 [ecj-lint] 3. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 182)
 [ecj-lint] c = getFromJndi(initProps, jndiName);
 [ecj-lint] ^^^
 [ecj-lint] The method getFromJndi(Properties, String) from the type new 
Callable(){} refers to the missing type NamingException
 [ecj-lint] --
 [ecj-lint] 4. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 245)
 [ecj-lint] private Connection getFromJndi(final Properties initProps, 
final String jndiName) throws NamingException,
 [ecj-lint] 
 ^^^
 [ecj-lint] NamingException cannot be resolved to a type
 [ecj-lint] --
 [ecj-lint] 5. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 249)
 [ecj-lint] InitialContext ctx =  new InitialContext();
 [ecj-lint] ^^
 [ecj-lint] InitialContext cannot be resolved to a type
 [ecj-lint] --
 [ecj-lint] 6. ERROR in 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/contrib/dataimporthandler/src/java/org/apache/solr/handler/dataimport/JdbcDataSource.java
 (at line 249)
 [ecj-lint] InitialContext ctx =  new InitialContext();
 [ecj-lint]   ^^
 [ecj-lint] InitialContext cannot be resolved to a type
 [ecj-lint] --
 [ecj-lint] 6 problems (6 errors)

BUILD FAILED
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:634: 
The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/build.xml:101: 
The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/build.xml:651:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/solr/common-build.xml:479:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2009:
 The following error occurred while executing this line:
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-master/lucene/common-build.xml:2048:
 Compile failed; see the compiler error output for details.

Total time: 238 minutes 54 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-13-ea+18) - Build # 24244 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/24244/
Java: 64bit/jdk-13-ea+18 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.NestedShardedAtomicUpdateTest.test

Error Message:
Error from server at http://127.0.0.1:38047/collection1: no servers hosting 
shard: shard1

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:38047/collection1: no servers hosting shard: 
shard1
at 
__randomizedtesting.SeedInfo.seed([DD0C36894EE69746:55580953E01AFABE]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:656)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:987)
at org.apache.solr.client.solrj.SolrClient.query(SolrClient.java:1002)
at 
org.apache.solr.cloud.NestedShardedAtomicUpdateTest.doNestedInplaceUpdateTest(NestedShardedAtomicUpdateTest.java:154)
at 
org.apache.solr.cloud.NestedShardedAtomicUpdateTest.test(NestedShardedAtomicUpdateTest.java:56)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  

[GitHub] [lucene-solr] rmuir commented on a change in pull request #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

rmuir commented on a change in pull request #722: LUCENE-8863: handle some edge 
cases in Kuromoji DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#discussion_r294475330
 
 

 ##
 File path: 
lucene/analysis/kuromoji/src/tools/java/org/apache/lucene/analysis/ja/util/BinaryDictionaryWriter.java
 ##
 @@ -126,7 +132,9 @@ public int put(String[] entry) {
 buffer.putShort(wordCost);
 
 if ((flags & BinaryDictionary.HAS_BASEFORM) != 0) {
-  assert baseForm.length() < 16;
+  if (baseForm.length() >= 16) {
 
 Review comment:
   I am referring to the exception message. this is one reason why the `assert` 
works well, such bugs are impossible, just as long as its run with `-ea` which 
the build file does :)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-8863) Improve handling of edge cases in Kuromoji's DIctionaryBuilder

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865906#comment-16865906
 ] 

Mike Sokolov edited comment on LUCENE-8863 at 6/17/19 7:10 PM:
---

OK, I will check for empty base form and raise an exception, not allow it. 
People can pass * if they want to have no base form. I think it is valid for 
any single one of the several POS fields in the input to be empty. We currently 
join them together with "-" separators unless they are empty, in which case we 
ignore them. I guess we could check if *none* of the POS fields have a 
non-empty value and throw an error in that case.


was (Author: sokolov):
OK, I will check for empty base form and raise an exception, not allow it. 
People can pass '{{*}}' if they want to have no base form. I think it is valid 
for any single one of the several POS fields in the input to be empty. We 
currently join them together with "-" separators unless they are empty, in 
which case we ignore them. I guess we could check if *none* of the POS fields 
have a non-empty value and throw an error in that case.

> Improve handling of edge cases in Kuromoji's DIctionaryBuilder
> --
>
> Key: LUCENE-8863
> URL: https://issues.apache.org/jira/browse/LUCENE-8863
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mike Sokolov
>Assignee: Mike Sokolov
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> While building a custom Kuromoji system dictionary, I discovered a few issues.
> First, the dictionary encoding has room for 13-bit (left and right) ids, but 
> really only supports 12 bits since this was all that was needed for the 
> IPADIC dictionary that ships with Kuromoji. The good news is we can easily 
> add support by fixing the bit-twiddling math.
> Second, the dictionary builder has a number of assertions that help uncover 
> problems in the input (like these overlarge ids), but the assertions aren't 
> enabled by default, so an unsuspecting new user doesn't get any benefit from 
> them, so we should upgrade to "real" exceptions.
> Finally, we want to handle the case of empty base forms differently. Kuromoji 
> does stemming by substituting a base form for a word when there is a base 
> form in the dictionary. Missing base forms are expected to be supplied as 
> {{*}}, but if a dictionary provides an empty string base form, we would end 
> up stripping that token completely. Since there is no possible meaning for an 
> empty base form (and the dictionary builder already treats {{*}} and empty 
> strings as equivalent in a number of other cases), I think we should simply 
> ignore empty base forms (rather than replacing words with empty strings when 
> tokenizing!)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13419) Add TRA denoting Infix to TRA collection names

[Gus Heck (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865907#comment-16865907
 ] 

Gus Heck commented on SOLR-13419:
-

Updated patch for current master, just some test readability changes and 
CHANGES.txt entry. Will commit soon since this has been avail for review for 
1.5 mo.

> Add TRA denoting Infix to TRA collection names
> --
>
> Key: SOLR-13419
> URL: https://issues.apache.org/jira/browse/SOLR-13419
> Project: Solr
>  Issue Type: Sub-task
>  Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13419.patch, SOLR-13419.patch
>
>
> One of the things that Dimensional Routed aliases (DRA) will need to be able 
> to do is provide the component routing features based on the route value in 
> the collection name. With multiple dimensions it becomes necessary to pull 
> this information out on a per-dimension basis and the current Time Routed 
> Alias (TRA) collection names have only a simple underscore between the alias 
> name and the route value timestamp. Furthermore the timestamps are variable 
> in length, having truncation of low significance zero fields as an 
> intentional feature (collections falling evenly on the month boundary will 
> omit the day/hour/min fields). This makes recognizing the TRA part of a DRA 
> complex.  This issue will bring TRA's in line with CRA's so that both follow 
> the pattern:
> alias__type__routevalue
> Specifically the new pattern will use __TRA__ for the type section. This 
> change must not break existing TRA's, and will not update existing collection 
> names, but the two formats have to co-exist peacefully going forward. DRA's 
> will only support time dimensions with the new format however, with a general 
> form of:
> alias__type__routevalue__type__routevalue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-8863) Improve handling of edge cases in Kuromoji's DIctionaryBuilder

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865906#comment-16865906
 ] 

Mike Sokolov edited comment on LUCENE-8863 at 6/17/19 7:08 PM:
---

OK, I will check for empty base form and raise an exception, not allow it. 
People can pass '{{*}}' if they want to have no base form. I think it is valid 
for any single one of the several POS fields in the input to be empty. We 
currently join them together with "-" separators unless they are empty, in 
which case we ignore them. I guess we could check if *none* of the POS fields 
have a non-empty value and throw an error in that case.


was (Author: sokolov):
OK, I will check for empty base form and raise an exception, not allow it. 
People can pass '*' if they want to have no base form. I think it is valid for 
any single one of the several POS fields in the input to be empty. We currently 
join them together with "-" separators unless they are empty, in which case we 
ignore them. I guess we could check if *none* of the POS fields have a 
non-empty value and throw an error in that case.

> Improve handling of edge cases in Kuromoji's DIctionaryBuilder
> --
>
> Key: LUCENE-8863
> URL: https://issues.apache.org/jira/browse/LUCENE-8863
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mike Sokolov
>Assignee: Mike Sokolov
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> While building a custom Kuromoji system dictionary, I discovered a few issues.
> First, the dictionary encoding has room for 13-bit (left and right) ids, but 
> really only supports 12 bits since this was all that was needed for the 
> IPADIC dictionary that ships with Kuromoji. The good news is we can easily 
> add support by fixing the bit-twiddling math.
> Second, the dictionary builder has a number of assertions that help uncover 
> problems in the input (like these overlarge ids), but the assertions aren't 
> enabled by default, so an unsuspecting new user doesn't get any benefit from 
> them, so we should upgrade to "real" exceptions.
> Finally, we want to handle the case of empty base forms differently. Kuromoji 
> does stemming by substituting a base form for a word when there is a base 
> form in the dictionary. Missing base forms are expected to be supplied as 
> {{*}}, but if a dictionary provides an empty string base form, we would end 
> up stripping that token completely. Since there is no possible meaning for an 
> empty base form (and the dictionary builder already treats {{*}} and empty 
> strings as equivalent in a number of other cases), I think we should simply 
> ignore empty base forms (rather than replacing words with empty strings when 
> tokenizing!)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8863) Improve handling of edge cases in Kuromoji's DIctionaryBuilder

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865906#comment-16865906
 ] 

Mike Sokolov commented on LUCENE-8863:
--

OK, I will check for empty base form and raise an exception, not allow it. 
People can pass '*' if they want to have no base form. I think it is valid for 
any single one of the several POS fields in the input to be empty. We currently 
join them together with "-" separators unless they are empty, in which case we 
ignore them. I guess we could check if *none* of the POS fields have a 
non-empty value and throw an error in that case.

> Improve handling of edge cases in Kuromoji's DIctionaryBuilder
> --
>
> Key: LUCENE-8863
> URL: https://issues.apache.org/jira/browse/LUCENE-8863
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mike Sokolov
>Assignee: Mike Sokolov
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> While building a custom Kuromoji system dictionary, I discovered a few issues.
> First, the dictionary encoding has room for 13-bit (left and right) ids, but 
> really only supports 12 bits since this was all that was needed for the 
> IPADIC dictionary that ships with Kuromoji. The good news is we can easily 
> add support by fixing the bit-twiddling math.
> Second, the dictionary builder has a number of assertions that help uncover 
> problems in the input (like these overlarge ids), but the assertions aren't 
> enabled by default, so an unsuspecting new user doesn't get any benefit from 
> them, so we should upgrade to "real" exceptions.
> Finally, we want to handle the case of empty base forms differently. Kuromoji 
> does stemming by substituting a base form for a word when there is a base 
> form in the dictionary. Missing base forms are expected to be supplied as 
> {{*}}, but if a dictionary provides an empty string base form, we would end 
> up stripping that token completely. Since there is no possible meaning for an 
> empty base form (and the dictionary builder already treats {{*}} and empty 
> strings as equivalent in a number of other cases), I think we should simply 
> ignore empty base forms (rather than replacing words with empty strings when 
> tokenizing!)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13419) Add TRA denoting Infix to TRA collection names

[Gus Heck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gus Heck updated SOLR-13419:

Attachment: SOLR-13419.patch

> Add TRA denoting Infix to TRA collection names
> --
>
> Key: SOLR-13419
> URL: https://issues.apache.org/jira/browse/SOLR-13419
> Project: Solr
>  Issue Type: Sub-task
>  Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13419.patch, SOLR-13419.patch
>
>
> One of the things that Dimensional Routed aliases (DRA) will need to be able 
> to do is provide the component routing features based on the route value in 
> the collection name. With multiple dimensions it becomes necessary to pull 
> this information out on a per-dimension basis and the current Time Routed 
> Alias (TRA) collection names have only a simple underscore between the alias 
> name and the route value timestamp. Furthermore the timestamps are variable 
> in length, having truncation of low significance zero fields as an 
> intentional feature (collections falling evenly on the month boundary will 
> omit the day/hour/min fields). This makes recognizing the TRA part of a DRA 
> complex.  This issue will bring TRA's in line with CRA's so that both follow 
> the pattern:
> alias__type__routevalue
> Specifically the new pattern will use __TRA__ for the type section. This 
> change must not break existing TRA's, and will not update existing collection 
> names, but the two formats have to co-exist peacefully going forward. DRA's 
> will only support time dimensions with the new format however, with a general 
> form of:
> alias__type__routevalue__type__routevalue



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] msokolov commented on a change in pull request #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

msokolov commented on a change in pull request #722: LUCENE-8863: handle some 
edge cases in Kuromoji DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#discussion_r294472350
 
 

 ##
 File path: 
lucene/analysis/kuromoji/src/tools/java/org/apache/lucene/analysis/ja/util/BinaryDictionaryWriter.java
 ##
 @@ -126,7 +132,9 @@ public int put(String[] entry) {
 buffer.putShort(wordCost);
 
 if ((flags & BinaryDictionary.HAS_BASEFORM) != 0) {
-  assert baseForm.length() < 16;
+  if (baseForm.length() >= 16) {
 
 Review comment:
   Could you elaborate? I do see the exception message is wrong (should say 
>=), but I think the assertion is correct" `len < 16` iff `!(len >= 16)`


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] msokolov commented on a change in pull request #722: LUCENE-8863: handle some edge cases in Kuromoji DictionaryBuilder, en…

[GitBox]

msokolov commented on a change in pull request #722: LUCENE-8863: handle some 
edge cases in Kuromoji DictionaryBuilder, en…
URL: https://github.com/apache/lucene-solr/pull/722#discussion_r294472510
 
 

 ##
 File path: 
lucene/analysis/kuromoji/src/tools/java/org/apache/lucene/analysis/ja/util/BinaryDictionaryWriter.java
 ##
 @@ -35,9 +35,11 @@
 import org.apache.lucene.analysis.ja.dict.BinaryDictionary;
 
 public abstract class BinaryDictionaryWriter {
+  private final static int ID_LIMIT = 8192;
+
   protected final Class implClazz;
   protected ByteBuffer buffer;
-  private int targetMapEndOffset = 0, lastWordId = -1, lastSourceId = -1;
+  protected int targetMapEndOffset = 0, lastWordId = -1, lastSourceId = -1;
 
 Review comment:
   my bad - this was left this way from some earlier debugging efforts. I'll 
undo


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-MAVEN] Lucene-Solr-Maven-8.x #129: POMs out of sync

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Maven-8.x/129/

No tests ran.

Build Log:
[...truncated 33497 lines...]
  [mvn] [INFO] -
  [mvn] [INFO] -
  [mvn] [ERROR] COMPILATION ERROR : 
  [mvn] [INFO] -

[...truncated 877 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-8.x/build.xml:680: The 
following error occurred while executing this line:
: Java returned: 1

Total time: 18 minutes 25 seconds
Build step 'Invoke Ant' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8859) Add an option to load the completion suggester's FST off-heap

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865884#comment-16865884
 ] 

Adrien Grand commented on LUCENE-8859:
--

I don't have objections to exposing this option. This makes me wonder whether 
we could generally make it easier to preload files, or slices, that we expect 
to be in the page cache with MMapDirectory.

> Add an option to load the completion suggester's FST off-heap
> -
>
> Key: LUCENE-8859
> URL: https://issues.apache.org/jira/browse/LUCENE-8859
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Jim Ferenczi
>Priority: Minor
> Attachments: LUCENE-8859.patch
>
>
> Now that FSTs can be loaded off-heap 
> (https://issues.apache.org/jira/browse/LUCENE-8635) it would be nice to 
> expose this option in the completion suggester postings format. I didn't ran 
> any benchmark yet so I can't say it this really makes sense or not but I 
> wanted to get some opinion whether this could be a good trade-off.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Atri Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865883#comment-16865883
 ] 

Atri Sharma commented on LUCENE-8862:
-

The reason I was thinking of a visitor api is since it gives a cleaner 
interface (not all children collectors need to expose the new constructor). 
Also, having the Visitor API gives the scope of more functionalities that the 
user could cook up (for eg, something like consumeTerms?)

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-master - Build # 1365 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1365/

No tests ran.

Build Log:
[...truncated 24664 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2581 links (2111 relative) to 3394 anchors in 259 files
 [echo] Validated Links & Anchors via: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/solr/package/solr-9.0.0.tgz
 into 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-master/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

Re: [VOTE] Release PyLucene 8.1.1

[Andi Vajda]

On Mon, 17 Jun 2019, David Allouche wrote:


Thank you, that was very informative.

+0 for this release, I builds and pass my test suite.

But I was unable to make a complete integration test because I do not have 
a proper index format migration infrastructure and my index is made of 
incompletely-upgraded lucene6 and lucene7 segments.


So far, you're the only one who has cast a vote.
No one else, no PMC member, none of the longtime users, no one.

Besides the Apache procedural aspect, voting is also a gauge of interest in 
a project. If there is no user interest in the PyLucene project anymore

maybe it's time to stop making releases for a while ?

PyLucene is a bit special in that it doesn't involve many people for 
development since it is machine-generated by JCC and JCC has been stable for 
a while. In that sense, it is important that actual users of PyLucene make 
themselves known by voting, not much else goes on on the project's mailing 
list or in the source code.


If there are actual users showing interest by voting here, I feel more 
confortable then in nagging people on the Lucene PMC for their procedural 
vote.


Andi..



I will post my questions about index upgrading on 
java-u...@lucene.apache.org .





Regards.


On 11 Jun 2019, at 16:50, Andi Vajda  wrote:



On Jun 11, 2019, at 06:30, David Allouche  wrote:

This is maybe a silly question, but what is the purpose of this voting process?


By the rules of Apache, three PMC binding votes are needed to make a release. 
In addition, it's a gauge of general interest in the project.


Is this something required by the project governance?


https://www.apache.org/foundation/voting.html
(see "votes on package releases", in particular)


What is the meaning of a vote? Does that mean "I am interested", or does it mean "I 
have tested the latest trunk and it looks good", or something else?


If you find a bug in the release artifacts (not in the latest trunk) before the 
release is made, the release is likely to be pulled.


What is the typical expected delay for reply? For example, I reserve Fridays 
for technical debt management (including upgrading dependencies), so I cannot 
typically validate a new PyLucene version in less than a week.


A vote must run for at least 72 hours.
Because you are not on the PMC, your vote falls into the "interest gauging" category, is 
not binding and is considered "best effort".


This is probably all common questions with well documented answers. If that's 
the case, then it would be nice to have a link to the answers in VOTE requests.


https://www.apache.org/foundation/voting.html

Andi..




On 11 Jun 2019, at 00:39, Andi Vajda  wrote:


The PyLucene 8.1.1 (rc1) release tracking the recent release of
Apache Lucene 8.1.1 is ready.

A release candidate is available from:
https://dist.apache.org/repos/dist/dev/lucene/pylucene/8.1.1-rc1/

PyLucene 8.1.1 is built with JCC 3.5, included in these release artifacts.

JCC 3.5 supports Python 3.3+ (in addition to Python 2.3+).
PyLucene may be built with Python 2 or Python 3.

Please vote to release these artifacts as PyLucene 8.1.1.
Anyone interested in this release can and should vote !

Thanks !

Andi..

ps: the KEYS file for PyLucene release signing is at:
https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS

pps: here is my +1

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865879#comment-16865879
 ] 

Adrien Grand commented on LUCENE-8862:
--

I see. Do we need a visitor at all, we could pass the counter to the 
constructor of the collector?

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Atri Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865873#comment-16865873
 ] 

Atri Sharma commented on LUCENE-8862:
-

Visitor could have an API to take in a memory tracking object (maybe as simple 
as an AtomicLong with some decorations). The patent collector passes it to 
down, and every supporting collector accepts that visitor and stores the object 
to update memory usage. Since the object is shared, it should be up to date at 
any point in time. Of course, we will have to make the updates and reads 
concurrent safe. Thoughts?

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jtibshirani commented on issue #715: LUCENE-7714 Add a range query that takes advantage of index sorting.

[GitBox]

jtibshirani commented on issue #715: LUCENE-7714 Add a range query that takes 
advantage of index sorting.
URL: https://github.com/apache/lucene-solr/pull/715#issuecomment-502800464
 
 
   > Instead of adding this alternative in the 
NumericDocValuesField#slowRangeQuery we could add an alternative query in the 
SortedNumericDocValuesRangeQuery that would be used if the condition to run in 
the sorted case are not met (multi-valued field or a different default value 
for instance).
   
   This makes the most sense to me in terms of design, I had actually 
considered it originally but wasn't totally sure about it!
   
   I pushed some new changes that move the logic to a new query 
`IndexSortSortedNumericDocValuesRangeQuery` in sandbox. I undid the changes to 
the existing `SortedNumericDocValuesRangeQuery`. For now, the new query only 
handles `SortedNumericDocValues` fields, since I think that is the most 
important use case and it was a bit tricky to add support for 
`NumericDocValues` as well.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865859#comment-16865859
 ] 

Adrien Grand commented on LUCENE-8862:
--

One you have a visitor API, how would you implement per-collector limits for 
memory usage?

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8632) XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo shapes

[Nicholas Knize (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865858#comment-16865858
 ] 

Nicholas Knize commented on LUCENE-8632:


Opened a PR for adding the ability to index and search non Geo / GIS Geometries 
by leveraging and abstracting much of the LatLonShape foundation. I chose the 
PR route since the number of lines is quite big and I figured it would be 
easier to review instead of attaching a patch.

> XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo 
> shapes
> -
>
> Key: LUCENE-8632
> URL: https://issues.apache.org/jira/browse/LUCENE-8632
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Nicholas Knize
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Currently the tessellator is tightly coupled with latitude and longitude 
> (WGS84) geospatial coordinates. This issue will explore generalizing the 
> tessellator, {{LatLonShape}} field and {{LatLonShapeQuery}} to non geospatial 
> (cartesian) coordinate systems so lucene can provide the index & search 
> capability for general geometry / non GIS type use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] nknize opened a new pull request #726: LUCENE-8632: New XYShape Field and Queries for indexing and searching general cartesian geometries

[GitBox]

nknize opened a new pull request #726: LUCENE-8632: New XYShape Field and 
Queries for indexing and searching general cartesian geometries
URL: https://github.com/apache/lucene-solr/pull/726
 
 
   The `LatLonShape` field and `LatLonShape` query classes added the ability to 
index and search geospatial geometries in the WGS-84 latitude, longitude 
coordinate reference system. The foundation for this capability is provided by 
the `Tessellator` that converts an array of vertices describing a `Point` 
`Line` or `Polygon` into a stream of 3 vertex triangles that are encoded as a 
seven dimension point and indexed using the BKD `POINT` structure. A nice 
property of the Tessellator is that `lat, lon` restrictions are artificial and 
really only bound by the API. 
   
   This commit builds on top of / abstracts the `Tessellator`  `LatLonShape` 
and `LatLonShapeQuery` classes to provide the ability to index & search general 
cartesian (non WGS84 lat,lon restricted) geometry. It does so by introducing 
two new base classes: `ShapeField` and `ShapeQuery` that provide the indexing 
and search foundation for `LatLonShape` and the `LatLonShape` derived query 
classes (`LatLonShapeBoundingBoxQuery`, `LatLonShapeLineQuery`, 
`LatLonShapePolygonQuery`) and introducing a new `XYShape` factory class along 
with `XYShape` derived query classes (`XYShapeBoundingBoxQuery`, 
`XYShapeLineQuery`, `XYShapePolygonQuery`). The heart of the cartesian indexing 
is achieved through `XYShapeEncodingUtils` that converts the double precision 
vertices into an `integer` encoded seven dimension point (similar to 
LatLonShape).
   
   The test framework is also further abstracted and extended to provide a full 
test suite for the new `XYShape` capability that works the same way as the 
`LatLonShape` test suite (but applied to non GIS geometries). 
   
   # Solution
   
   Please provide a short description of the approach taken to implement your 
solution.
   
   # Tests
   
   Please describe the tests you've developed or run to confirm this patch 
implements the feature or solves the problem.
   
   
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8632) XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo shapes

[Nicholas Knize (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Knize updated LUCENE-8632:
---
Description: Currently the tessellator is tightly coupled with latitude and 
longitude (WGS84) geospatial coordinates. This issue will explore generalizing 
the tessellator, {{LatLonShape}} field and {{LatLonShapeQuery}} to non 
geospatial (cartesian) coordinate systems so lucene can provide the index & 
search capability for general geometry / non GIS type use cases.  (was: 
Currently the tessellator is tightly coupled with latitude and longitude 
(WGS84) geospatial coordinates. This issue will explore generalizing the 
tessellator to non geospatial coordinate systems so lucene can handle arbitrary 
projections in vector space.)

> XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo 
> shapes
> -
>
> Key: LUCENE-8632
> URL: https://issues.apache.org/jira/browse/LUCENE-8632
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Nicholas Knize
>Priority: Major
>
> Currently the tessellator is tightly coupled with latitude and longitude 
> (WGS84) geospatial coordinates. This issue will explore generalizing the 
> tessellator, {{LatLonShape}} field and {{LatLonShapeQuery}} to non geospatial 
> (cartesian) coordinate systems so lucene can provide the index & search 
> capability for general geometry / non GIS type use cases.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8632) XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo shapes

[Nicholas Knize (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Knize updated LUCENE-8632:
---
Summary: XYShape: Adapt LatLonShape tessellator, field type, and queries to 
non-geo shapes  (was: Adapt LatLonShape tessellator, field type, and queries to 
non-geo shapes)

> XYShape: Adapt LatLonShape tessellator, field type, and queries to non-geo 
> shapes
> -
>
> Key: LUCENE-8632
> URL: https://issues.apache.org/jira/browse/LUCENE-8632
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Nicholas Knize
>Priority: Major
>
> Currently the tessellator is tightly coupled with latitude and longitude 
> (WGS84) geospatial coordinates. This issue will explore generalizing the 
> tessellator to non geospatial coordinate systems so lucene can handle 
> arbitrary projections in vector space.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8632) Adapt LatLonShape tessellator, field type, and queries to non-geo shapes

[Nicholas Knize (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nicholas Knize updated LUCENE-8632:
---
Summary: Adapt LatLonShape tessellator, field type, and queries to non-geo 
shapes  (was: Adapt LatLonShape tessellator to non-geo shapes)

> Adapt LatLonShape tessellator, field type, and queries to non-geo shapes
> 
>
> Key: LUCENE-8632
> URL: https://issues.apache.org/jira/browse/LUCENE-8632
> Project: Lucene - Core
>  Issue Type: New Feature
>Reporter: Nicholas Knize
>Assignee: Nicholas Knize
>Priority: Major
>
> Currently the tessellator is tightly coupled with latitude and longitude 
> (WGS84) geospatial coordinates. This issue will explore generalizing the 
> tessellator to non geospatial coordinate systems so lucene can handle 
> arbitrary projections in vector space.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-8.x - Build # 243 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/243/

All tests passed

Build Log:
[...truncated 64796 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/build.xml:634: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/build.xml:101: The 
following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/solr/build.xml:644: 
The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/lucene/common-build.xml:2093:
 The following error occurred while executing this line:
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Tests-8.x/lucene/common-build.xml:2132:
 Compile failed; see the compiler error output for details.

Total time: 110 minutes 4 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Recording test results
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865837#comment-16865837
 ] 

ASF subversion and git services commented on SOLR-13490:


Commit 2f2333a7814876871873cf050de8439c0c681c91 in lucene-solr's branch 
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2f2333a ]

SOLR-13490: Fix CollectionStateWatcher/CollectionStatePredicate based APIs in 
ZkStateReader and CloudSolrClient to be triggered on liveNode changes.

Also add Predicate equivilents for callers that don't care about 
liveNodes.

(cherry picked from commit 5a974860fa83408a86ca64b417f3111b037da7eb)


> waitForState/registerCollectionStateWatcher can see stale liveNodes data due 
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, 
> SOLR-13490.patch
>
>
> I was investigating some failures in 
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the 
> hunch that {{waitForState}} wasn't ensuring that the predicates registered 
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way 
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in 
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only 
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to 
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / 
> {{CollectionStatePredicate}} are called, they get the "fresh" 
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only 
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - 
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas 
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means 
> there is a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if 
> "replicaX" of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new 
> DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and 
> doesn't take any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified 
> fired with the updated liveNodes (unless/until another {{state.json}} change 
> is made for collectionA.
> 
> While this is definitely be problematic in _tests_ that deal with node 
> lifecyle and use things like {{SolrCloudTestCase.waitForState(..., 
> SolrCloudTestCase.clusterShape(...))}} to check for the expected 
> shards/replicas, a cursory search of how/where 
> {{ZkStateReader.waitForState(...)}} and 
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core 
> suggests that could also lead to bad behavior in situations like reacting to 
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for 
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by 
> the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865831#comment-16865831
 ] 

Munendra S N commented on SOLR-7530:


 [^SOLR-7530.patch] 
* I have modified changes entry so that arrays/namedlist is included. Let me 
know if any changes required here
* I tried keeping static response parser array and it works. I went through 
some parsers code doesn't look like stateful but to be on safer side extracted 
and added util method. 


> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N updated SOLR-7530:
---
Attachment: SOLR-7530.patch

> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch, SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>   "RTF":303
>   } 
> } } 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Atri Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865809#comment-16865809
 ] 

Atri Sharma commented on LUCENE-8862:
-

Thinking more about this, I am more inclined to do the Visitor approach for 
Collectors, since that gives us more functionality in terms of what can be done 
with the visitor, and memory accounting can be one aspect where it is used.

Unless there are objections, I am planning to work on a patch for initial 
review.

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-12) - Build # 8001 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/8001/
Java: 64bit/jdk-12 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader

Error Message:
Didn't get expected doc count. Expected: 10, Found: 0

Stack Trace:
java.lang.AssertionError: Didn't get expected doc count. Expected: 10, Found: 0
at 
__randomizedtesting.SeedInfo.seed([CEA95A5985ED571D:40A53A29F0F5D337]:0)
at org.junit.Assert.fail(Assert.java:88)
at 
org.apache.solr.cloud.TestPullReplicaErrorHandling.assertNumDocs(TestPullReplicaErrorHandling.java:252)
at 
org.apache.solr.cloud.TestPullReplicaErrorHandling.assertNumDocs(TestPullReplicaErrorHandling.java:257)
at 
org.apache.solr.cloud.TestPullReplicaErrorHandling.testCantConnectToLeader(TestPullReplicaErrorHandling.java:196)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)




Build Log:
[...truncated 14498 lines...]
   [junit4] 

[GitHub] [lucene-solr] jpountz commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

jpountz commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294398016
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -636,16 +636,45 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   query = rewrite(query);
   final Weight weight = createWeight(query, scoreMode, 1);
   final List> topDocsFutures = new 
ArrayList<>(leafSlices.length);
+
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length-1) { // execute the last on the caller 
thread
+  search(Arrays.asList(leaves), weight, collector);
+  topDocsFutures.add(new Future<>() {
 
 Review comment:
   use `CompletableFuture#completedFuture`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] s1monw commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

s1monw commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294396310
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -636,16 +636,45 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   query = rewrite(query);
   final Weight weight = createWeight(query, scoreMode, 1);
   final List> topDocsFutures = new 
ArrayList<>(leafSlices.length);
+
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length-1) { // execute the last on the caller 
thread
+  search(Arrays.asList(leaves), weight, collector);
+  topDocsFutures.add(new Future<>() {
+
+@Override
+public boolean cancel(boolean mayInterruptIfRunning) {
+  return false;
+}
+
+@Override
+public boolean isCancelled() {
+  return false;
+}
+
+@Override
+public boolean isDone() {
+  return true;
+}
+
+@Override
+public C get() {
+  return collector;
+}
+
+@Override
+public C get(long timeout, TimeUnit unit) {
+  return get();
+}
 
 Review comment:
   well I looked at this code accidentally investigating some idea and I 
figured it doesn't make sense to just block and wait instead of executing a 
search so if you are having a single segment you never pay the price of 
spawning a thread and you don't do any unnecessary work.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865773#comment-16865773
 ] 

ASF subversion and git services commented on SOLR-13490:


Commit 5a974860fa83408a86ca64b417f3111b037da7eb in lucene-solr's branch 
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5a97486 ]

SOLR-13490: Fix CollectionStateWatcher/CollectionStatePredicate based APIs in 
ZkStateReader and CloudSolrClient to be triggered on liveNode changes.

Also add Predicate equivilents for callers that don't care about 
liveNodes.


> waitForState/registerCollectionStateWatcher can see stale liveNodes data due 
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
>  Issue Type: Bug
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Attachments: SOLR-13490.patch, SOLR-13490.patch, SOLR-13490.patch, 
> SOLR-13490.patch
>
>
> I was investigating some failures in 
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the 
> hunch that {{waitForState}} wasn't ensuring that the predicates registered 
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way 
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in 
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only 
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to 
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} / 
> {{CollectionStatePredicate}} are called, they get the "fresh" 
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only 
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} - 
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas 
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means 
> there is a race condition where the following can happen...
>  * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
>  * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if 
> "replicaX" of collectionA is on a "down" node
>  * client2 causes shutdown of node N1 which is hosting replicaX
>  * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
>  ** DocCollection for collectionA is rebuilt
>  ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new 
> DocCollection
>  *** watcherZ sees that replicaX is on N1, but thinks N1 is live
>  *** watcherZ says "everything ok, not the event i was waiting for" and 
> doesn't take any action
>  * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
>  ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified 
> fired with the updated liveNodes (unless/until another {{state.json}} change 
> is made for collectionA.
> 
> While this is definitely be problematic in _tests_ that deal with node 
> lifecyle and use things like {{SolrCloudTestCase.waitForState(..., 
> SolrCloudTestCase.clusterShape(...))}} to check for the expected 
> shards/replicas, a cursory search of how/where 
> {{ZkStateReader.waitForState(...)}} and 
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core 
> suggests that could also lead to bad behavior in situations like reacting to 
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for 
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by 
> the watcher/predicate)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk-12) - Build # 721 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/721/
Java: 64bit/jdk-12 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.NestedShardedAtomicUpdateTest.test

Error Message:
Error from server at http://127.0.0.1:35477/a/collection1: non ok status: 500, 
message:Server Error

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:35477/a/collection1: non ok status: 500, 
message:Server Error
at 
__randomizedtesting.SeedInfo.seed([74699B92B3C6BF62:FC3DA4481D3AD29A]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:586)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:228)
at 
org.apache.solr.BaseDistributedSearchTestCase.add(BaseDistributedSearchTestCase.java:576)
at 
org.apache.solr.cloud.NestedShardedAtomicUpdateTest.indexDocAndRandomlyCommit(NestedShardedAtomicUpdateTest.java:221)
at 
org.apache.solr.cloud.NestedShardedAtomicUpdateTest.sendWrongRouteParam(NestedShardedAtomicUpdateTest.java:191)
at 
org.apache.solr.cloud.NestedShardedAtomicUpdateTest.test(NestedShardedAtomicUpdateTest.java:55)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1082)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1054)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[GitHub] [lucene-solr] atris commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

atris commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294377314
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -636,16 +636,45 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   query = rewrite(query);
   final Weight weight = createWeight(query, scoreMode, 1);
   final List> topDocsFutures = new 
ArrayList<>(leafSlices.length);
+
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length-1) { // execute the last on the caller 
thread
 
 Review comment:
   Nit: Spacing


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] atris commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

atris commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294377416
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -636,16 +636,45 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   query = rewrite(query);
   final Weight weight = createWeight(query, scoreMode, 1);
   final List> topDocsFutures = new 
ArrayList<>(leafSlices.length);
+
 
 Review comment:
   Nit: Unnecessary newline


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] atris commented on a change in pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

atris commented on a change in pull request #725: LUCENE-8865: Use incoming 
thread for execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#discussion_r294382238
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/IndexSearcher.java
 ##
 @@ -636,16 +636,45 @@ public TopFieldDocs reduce(Collection 
collectors) throws IOEx
   query = rewrite(query);
   final Weight weight = createWeight(query, scoreMode, 1);
   final List> topDocsFutures = new 
ArrayList<>(leafSlices.length);
+
   for (int i = 0; i < leafSlices.length; ++i) {
 final LeafReaderContext[] leaves = leafSlices[i].leaves;
 final C collector = collectors.get(i);
-topDocsFutures.add(executor.submit(new Callable() {
-  @Override
-  public C call() throws Exception {
+if (i == leafSlices.length-1) { // execute the last on the caller 
thread
+  search(Arrays.asList(leaves), weight, collector);
+  topDocsFutures.add(new Future<>() {
+
+@Override
+public boolean cancel(boolean mayInterruptIfRunning) {
+  return false;
+}
+
+@Override
+public boolean isCancelled() {
+  return false;
+}
+
+@Override
+public boolean isDone() {
+  return true;
+}
+
+@Override
+public C get() {
+  return collector;
+}
+
+@Override
+public C get(long timeout, TimeUnit unit) {
+  return get();
+}
 
 Review comment:
   Nice Change! I am curious to understand though -- did we see this as an 
issue, or did we think of this as an optimization? Of course, that is not a 
blocker to this change


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7530) Wrong JSON response using Terms Component with distrib=true

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865738#comment-16865738
 ] 

Lucene/Solr QA commented on SOLR-7530:
--

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 2 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
31s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  2m 
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  2m 14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  2m  7s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  2m  7s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}149m 29s{color} 
| {color:red} core in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green}  0m 
33s{color} | {color:green} test-framework in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}158m  9s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.cloud.CollectionPropsTest |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-7530 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12971950/SOLR-7530.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / dff7611 |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| unit | 
https://builds.apache.org/job/PreCommit-SOLR-Build/440/artifact/out/patch-unit-solr_core.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/440/testReport/ |
| modules | C: solr solr/core solr/test-framework U: solr |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/440/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> Wrong JSON response using Terms Component with distrib=true
> ---
>
> Key: SOLR-7530
> URL: https://issues.apache.org/jira/browse/SOLR-7530
> Project: Solr
>  Issue Type: Bug
>  Components: Response Writers, SearchComponents - other, SolrCloud
>Affects Versions: 4.9
>Reporter: Raúl Grande
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: SOLR-7530.patch, SOLR-7530.patch, SOLR-7530.patch, 
> SOLR-7530.patch
>
>
> When using TermsComponent in SolrCloud there are differences in the JSON 
> response if parameter distrib is true or false. If distrib=true JSON is not 
> well formed (please note at the [ ] marks)
> JSON Response when distrib=false. Correct response:
> {"responseHeader":{ 
>   "status":0, 
>   "QTime":3
> }, 
> "terms":{ 
> "FileType":
> [ 
>   "EMAIL",20060, 
>   "PDF",7051, 
>   "IMAGE",5108, 
>   "OFFICE",4912, 
>   "TXT",4405, 
>   "OFFICE_EXCEL",4122, 
>   "OFFICE_WORD",2468
>   ]
> } } 
> JSON Response when distrib=true. Incorrect response:
> { 
> "responseHeader":{
>   "status":0, 
>   "QTime":94
> }, 
> "terms":{ 
> "FileType":{ 
>   "EMAIL":31923, 
>   "PDF":11545, 
>   "IMAGE":9807, 
>   "OFFICE_EXCEL":8195, 
>   "OFFICE":5147, 
>   "OFFICE_WORD":4820, 
>   "TIFF":1156, 
>   "XML":851, 
>   "HTML":821, 
>  

[GitHub] [lucene-solr] s1monw commented on issue #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

s1monw commented on issue #725: LUCENE-8865: Use incoming thread for execution 
if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725#issuecomment-502747988
 
 
   @mikemccand can you take a look


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] s1monw opened a new pull request #725: LUCENE-8865: Use incoming thread for execution if IndexSearcher has an executor

[GitBox]

s1monw opened a new pull request #725: LUCENE-8865: Use incoming thread for 
execution if IndexSearcher has an executor
URL: https://github.com/apache/lucene-solr/pull/725
 
 
   Today we don't utilize the incoming thread for a search when IndexSearcher
   has an executor. This thread is only idling but can be used to execute a 
search
   once all other collectors are dispatched.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8865) Use incoming thread for execution if IndexSearcher has an executor

[Simon Willnauer (JIRA)]

Simon Willnauer created LUCENE-8865:
---

 Summary:  Use incoming thread for execution if IndexSearcher has 
an executor
 Key: LUCENE-8865
 URL: https://issues.apache.org/jira/browse/LUCENE-8865
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Simon Willnauer


Today we don't utilize the incoming thread for a search when IndexSearcher
has an executor. This thread is only idleing but can be used to execute a 
search
once all other collectors are dispatched.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8855) Add Accountable to Query implementations

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865699#comment-16865699
 ] 

Andrzej Bialecki  commented on LUCENE-8855:
---

[~jpountz] - it's true that in most cases nobody cares about Query memory 
footprint, but the caching use case is an important one, and it needs more 
support from the API in order to better estimate the actual cost of caching.

I don't think using Weight#isCacheable helps in this particular scenario 
because you need to calculate Weight first, which brings its own significant 
cost.

QueryVisitor API is an option, sure, but still the Query implementation itself 
knows the best what its memory use is like, so it's still the Query impl that 
needs to calculate this estimate - and then report it, either using a method 
that's an extension of QueryVisitor or the already available Accountable.

> Add Accountable to Query implementations
> 
>
> Key: LUCENE-8855
> URL: https://issues.apache.org/jira/browse/LUCENE-8855
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
> Attachments: LUCENE-8855.patch, LUCENE-8855.patch
>
>
> Query implementations should also support {{Accountable}} API in order to 
> monitor the memory consumption e.g. in caches where either keys or values are 
> {{Query}} instances.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1873 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1873/

7 tests failed.
FAILED:  
org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat.testByteNumericsVsStoredFields

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([8682CDFBDF599718]:0)


FAILED:  
junit.framework.TestSuite.org.apache.lucene.codecs.perfield.TestPerFieldDocValuesFormat

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([8682CDFBDF599718]:0)


FAILED:  
org.apache.lucene.search.TestSimpleSearchEquivalence.testBooleanBoostPropagation

Error Message:
expected:<5577> but was:<5622>

Stack Trace:
java.lang.AssertionError: expected:<5577> but was:<5622>
at 
__randomizedtesting.SeedInfo.seed([8682CDFBDF599718:AACF0A8C3EBDD1C3]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at 
org.apache.lucene.search.SearchEquivalenceTestBase.assertSameScores(SearchEquivalenceTestBase.java:255)
at 
org.apache.lucene.search.SearchEquivalenceTestBase.assertSameScores(SearchEquivalenceTestBase.java:228)
at 
org.apache.lucene.search.TestSimpleSearchEquivalence.testBooleanBoostPropagation(TestSimpleSearchEquivalence.java:235)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

Re: [VOTE] Release PyLucene 8.1.1

[David Allouche]

Thank you, that was very informative.

+0 for this release, I builds and pass my test suite.

But I was unable to make a complete integration test because I do not have a 
proper index format migration infrastructure and my index is made of 
incompletely-upgraded lucene6 and lucene7 segments.

I will post my questions about index upgrading on java-u...@lucene.apache.org 
.

Regards.

> On 11 Jun 2019, at 16:50, Andi Vajda  wrote:
> 
> 
>> On Jun 11, 2019, at 06:30, David Allouche  wrote:
>> 
>> This is maybe a silly question, but what is the purpose of this voting 
>> process?
> 
> By the rules of Apache, three PMC binding votes are needed to make a release. 
> In addition, it's a gauge of general interest in the project.
> 
>> Is this something required by the project governance?
> 
> https://www.apache.org/foundation/voting.html
> (see "votes on package releases", in particular)
> 
>> What is the meaning of a vote? Does that mean "I am interested", or does it 
>> mean "I have tested the latest trunk and it looks good", or something else?
> 
> If you find a bug in the release artifacts (not in the latest trunk) before 
> the release is made, the release is likely to be pulled.
> 
>> What is the typical expected delay for reply? For example, I reserve Fridays 
>> for technical debt management (including upgrading dependencies), so I 
>> cannot typically validate a new PyLucene version in less than a week.
> 
> A vote must run for at least 72 hours.
> Because you are not on the PMC, your vote falls into the "interest gauging" 
> category, is not binding and is considered "best effort".
> 
>> This is probably all common questions with well documented answers. If 
>> that's the case, then it would be nice to have a link to the answers in VOTE 
>> requests.
> 
> https://www.apache.org/foundation/voting.html
> 
> Andi..
> 
>> 
>>> On 11 Jun 2019, at 00:39, Andi Vajda  wrote:
>>> 
>>> 
>>> The PyLucene 8.1.1 (rc1) release tracking the recent release of
>>> Apache Lucene 8.1.1 is ready.
>>> 
>>> A release candidate is available from:
>>> https://dist.apache.org/repos/dist/dev/lucene/pylucene/8.1.1-rc1/
>>> 
>>> PyLucene 8.1.1 is built with JCC 3.5, included in these release artifacts.
>>> 
>>> JCC 3.5 supports Python 3.3+ (in addition to Python 2.3+).
>>> PyLucene may be built with Python 2 or Python 3.
>>> 
>>> Please vote to release these artifacts as PyLucene 8.1.1.
>>> Anyone interested in this release can and should vote !
>>> 
>>> Thanks !
>>> 
>>> Andi..
>>> 
>>> ps: the KEYS file for PyLucene release signing is at:
>>> https://dist.apache.org/repos/dist/release/lucene/pylucene/KEYS
>>> https://dist.apache.org/repos/dist/dev/lucene/pylucene/KEYS
>>> 
>>> pps: here is my +1
>> 

BadApples this week:

[Erick Erickson]

Once again this week I won’t be changing any annotations.

According to Hoss’ rollups, SharedFSAutoReplicaFailoverTest is failing 100% of 
the time, it’s a nightly and it’s failure rate is 3/3

Full output attached.

**Annotated tests that didn't fail in the last 4 weeks.

  **Tests removed from the next two lists because they were specified in 
'doNotEnable' in the properties file
 MoveReplicaHDFSTest.testNormalFailedMove

  **Annotations will be removed from the following tests because they haven't 
failed in the last 4 rollups.

  **Methods: 3
   FullSolrCloudDistribCmdsTest.test
   SolrZkClientTest.testSimpleUpdateACLs
   TestCollectionStateWatchers.testCanWaitForNonexistantCollection


Failures in Hoss' reports for the last 4 rollups.

There were 530 unannotated tests that failed in Hoss' rollups. Ordered by the 
date I downloaded the rollup file, newest->oldest. See above for the dates the 
files were collected 
These tests were NOT BadApple'd or AwaitsFix'd
All tests that failed 4 weeks running will be BadApple'd unless there are 
objections

Failures in the last 4 reports..
   Report   Pct runsfails   test
 0123   4.3  944 34  
AliasIntegrationTest.testClusterStateProviderAPI
 0123   0.8  960204  BasicAuthIntegrationTest.testBasicAuth
 0123  27.3   87 21  
HdfsAutoAddReplicasIntegrationTest.testSimple
 0123   1.5  902 10  HttpPartitionTest.test
 0123   1.1  918 15  NestedShardedAtomicUpdateTest.test
 0123   0.8  907 10  PeerSyncReplicationTest.test
 0123   0.4  896  4  ReindexCollectionTest.testBasicReindexing
 0123   0.7  942  6  ShardSplitTest.testSplitShardWithRule
 0123  12.0   83  7  ShardSplitTest.testSplitWithChaosMonkey
 0123   1.1  883  6  SystemCollectionCompatTest.testBackCompat
 0123   0.7  897  5  TestDynamicLoading.testDynamicLoading
 0123   1.9  908 19  TestRegexpRandom2.testRegexps
 Will BadApple all tests above this line except ones listed at the 
top**

DO NOT ENABLE LIST:
MoveReplicaHDFSTest.testFailedMove
MoveReplicaHDFSTest.testNormalFailedMove
TestControlledRealTimeReopenThread.testCRTReopen
TestICUNormalizer2CharFilter.testRandomStrings
TestICUTokenizerCJK
TestImpersonationWithHadoopAuth.testForwarding
TestLTRReRankingPipeline.testDifferentTopN
TestRandomChains


DO NOT ANNOTATE LIST
CdcrBidirectionalTest.testBiDir
IndexSizeTriggerTest.testMergeIntegration
IndexSizeTriggerTest.testMixedBounds
IndexSizeTriggerTest.testSplitIntegration
IndexSizeTriggerTest.testTrigger
InfixSuggestersTest.testShutdownDuringBuild
ShardSplitTest.test
ShardSplitTest.testSplitMixedReplicaTypes
ShardSplitTest.testSplitWithChaosMonkey
TestLatLonShapeQueries.testRandomBig
TestRandomChains.testRandomChainsWithLargeStrings
TestTriggerIntegration.testSearchRate

Processing file (History bit 3): HOSS-2019-06-17.csv
Processing file (History bit 2): HOSS-2019-06-10.csv
Processing file (History bit 1): HOSS-2019-06-03.csv
Processing file (History bit 0): HOSS-2019-05-28.csv


**Annotated tests that didn't fail in the last 4 weeks.

  **Tests removed from the next two lists because they were specified in 
'doNotEnable' in the properties file
 MoveReplicaHDFSTest.testNormalFailedMove

  **Annotations will be removed from the following tests because they haven't 
failed in the last 4 rollups.

  **Methods: 3
   FullSolrCloudDistribCmdsTest.test
   SolrZkClientTest.testSimpleUpdateACLs
   TestCollectionStateWatchers.testCanWaitForNonexistantCollection


Failures in Hoss' reports for the last 4 rollups.

There were 530 unannotated tests that failed in Hoss' rollups. Ordered by the 
date I downloaded the rollup file, newest->oldest. See above for the dates the 
files were collected 
These tests were NOT BadApple'd or AwaitsFix'd
All tests that failed 4 weeks running will be BadApple'd unless there are 
objections

Failures in the last 4 reports..
   Report   Pct runsfails   test
 0123   4.3  944 34  
AliasIntegrationTest.testClusterStateProviderAPI
 0123   0.8  960204  BasicAuthIntegrationTest.testBasicAuth
 0123  27.3   87 21  
HdfsAutoAddReplicasIntegrationTest.testSimple
 0123   1.5  902 10  HttpPartitionTest.test
 0123   1.1  918 15  NestedShardedAtomicUpdateTest.test
 0123   0.8  907 10  PeerSyncReplicationTest.test
 0123   0.4  896  4  ReindexCollectionTest.testBasicReindexing
 0123   0.7  942  6  ShardSplitTest.testSplitShardWithRule
 0123  12.0   83  7  ShardSplitTest.testSplitWithChaosMonkey
 0123   1.1  883  6  SystemCollectionCompatTest.testBackCompat
 0123   0.7  897  5  

[JENKINS] Lucene-Solr-8.x-Solaris (64bit/jdk1.8.0) - Build # 185 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/185/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth

Error Message:
Expected metric minimums for prefix SECURITY./authentication.: 
{failMissingCredentials=2, authenticated=19, passThrough=9, 
failWrongCredentials=1, requests=31, errors=0}, but got: 
{failMissingCredentials=2, authenticated=16, passThrough=13, totalTime=3937910, 
failWrongCredentials=1, requestTimes=1350, requests=32, errors=0}

Stack Trace:
java.lang.AssertionError: Expected metric minimums for prefix 
SECURITY./authentication.: {failMissingCredentials=2, authenticated=19, 
passThrough=9, failWrongCredentials=1, requests=31, errors=0}, but got: 
{failMissingCredentials=2, authenticated=16, passThrough=13, totalTime=3937910, 
failWrongCredentials=1, requestTimes=1350, requests=32, errors=0}
at 
__randomizedtesting.SeedInfo.seed([77E18C23451FFEAA:CB8FFA31E14C7DD0]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.solr.cloud.SolrCloudAuthTestCase.assertAuthMetricsMinimums(SolrCloudAuthTestCase.java:129)
at 
org.apache.solr.cloud.SolrCloudAuthTestCase.assertAuthMetricsMinimums(SolrCloudAuthTestCase.java:83)
at 
org.apache.solr.security.BasicAuthIntegrationTest.testBasicAuth(BasicAuthIntegrationTest.java:306)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 

Re: VOTE: Apache Solr Reference Guide for Solr 8.1

[Cassandra Targett]

Thanks everyone, this vote has passed and I’ll start the release process now.

Cassandra
On Jun 17, 2019, 7:51 AM -0500, Ishan Chattopadhyaya 
, wrote:
> +1
>
> > On Mon, 17 Jun, 2019, 5:46 PM Jason Gerlowski,  
> > wrote:
> > > +1 LGTM
> > >
> > > On Fri, Jun 14, 2019 at 5:44 PM Gus Heck  wrote:
> > > >
> > > > Noticed one thing that probably isn't enough to re-spin, but I'll fix 
> > > > for future:
> > > >
> > > > on Aliases page
> > > >
> > > > an alias must not refer to a Routed Alias (see below)
> > > >
> > > > shows up at the bottom of the page, and should say (see above) or omit 
> > > > the parentheses entirely.
> > > >
> > > > On Fri, Jun 14, 2019 at 5:32 PM Đạt Cao Mạnh  
> > > > wrote:
> > > >>
> > > >> +1
> > > >>
> > > >> On Fri, Jun 14, 2019 at 9:39 PM Jan Høydahl  
> > > >> wrote:
> > > >>>
> > > >>> +1
> > > >>>
> > > >>> Jan Høydahl
> > > >>>
> > > >>> 12. jun. 2019 kl. 16:47 skrev Cassandra Targett :
> > > >>>
> > > >>> Please vote to release the Solr Reference Guide for 8.1
> > > >>>
> > > >>> The PDF artifacts can be downloaded from:
> > > >>> https://dist.apache.org/repos/dist/dev/lucene/solr/ref-guide/apache-solr-ref-guide-8.1-RC1/
> > > >>>
> > > >>> $ cat apache-solr-ref-guide-8.1.pdf.sha512
> > > >>> cc76882fb3061fa03d1aa291d9705c1df17f948ff47f3f7d6a18e8ddef907f1c74f078ed482f7f5e04b7c6779a88ad85297cd31ae03570db2acc5930ba2feaf0
> > > >>>   apache-solr-ref-guide-8.1.pdf
> > > >>>
> > > >>> The HTML version is also available: 
> > > >>> https://lucene.apache.org/solr/guide/8_1
> > > >>>
> > > >>> Here's my +1.
> > > >>>
> > > >>> Thanks,
> > > >>> Cassandra
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Best regards,
> > > >> Cao Mạnh Đạt
> > > >> E-mail: caomanhdat...@gmail.com
> > > >
> > > >
> > > >
> > > > --
> > > > http://www.needhamsoftware.com (work)
> > > > http://www.the111shift.com (play)
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> > > For additional commands, e-mail: dev-h...@lucene.apache.org
> > >

[jira] [Commented] (LUCENE-8862) Collector Level Dynamic Memory Accounting

[Atri Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865610#comment-16865610
 ] 

Atri Sharma commented on LUCENE-8862:
-

{quote}Memory accounting has a cost API-wise
{quote}
Agreed, I was thinking of adding it as a default method which is essentially a 
no-op.

 
{quote}The only collectors I have in mind that might allocate lots of memory 
are the ones that are used to collect matches in a bitset
{quote}
Yeah, that is true. However, my bigger objective is to allow a generic 
interface for consuming and using this data (wherever it is exposed). For eg, I 
was thinking of adding a Collector which can wrap other Collectors and provide 
a way to fence memory usage.

 

Other way to think about this is if we should add a similar mechanism like 
QueryVisitor to Collectors, and then pursue an effort to add memory collection 
heuristics to both QueryVisitor and the Collector visitor. Having the Collector 
visitor interface will provide ways to have deeper functionality than just the 
memory accounting. WDYT?

> Collector Level Dynamic Memory Accounting
> -
>
> Key: LUCENE-8862
> URL: https://issues.apache.org/jira/browse/LUCENE-8862
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>
> Inspired from LUCENE-8855, I am thinking of adding a new interface which 
> tracks dynamic memory used by Collectors. This shall allow users to get an 
> accountability as to the memory usage of their Collectors and better plan 
> their resource capacity. This shall also allow us to add Collector level 
> limits for memory usage, thus allowing users a finer control over their 
> resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] ctargett commented on issue #643: SOLR-13391: Add variance and standard deviation stream evaluators

[GitBox]

ctargett commented on issue #643: SOLR-13391: Add variance and standard 
deviation stream evaluators
URL: https://github.com/apache/lucene-solr/pull/643#issuecomment-502683748
 
 
   @joel-bernstein Have you linked your GH and ASF accounts? It's a simple 
procedure of adding your GH ID to your ASF profile: 
https://reference.apache.org/committer/github. Once its sync'd you should be 
able to fully edit/merge PRs.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-MAVEN] Lucene-Solr-Maven-master #2588: POMs out of sync

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Maven-master/2588/

No tests ran.

Build Log:
[...truncated 35058 lines...]
BUILD FAILED
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-Maven-master/build.xml:680: The 
following error occurred while executing this line:
: Java returned: 1

Total time: 23 minutes 5 seconds
Build step 'Invoke Ant' marked build as failure
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13553) Node level custom RequestHandlers

[Noble Paul (JIRA)]

Noble Paul created SOLR-13553:
-

 Summary: Node level custom RequestHandlers
 Key: SOLR-13553
 URL: https://issues.apache.org/jira/browse/SOLR-13553
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Noble Paul


These components 
* Available on every node
* deployed at the {{CoreContainer}} level
* Available at {{/solr/admin/ or {{/api/node/}} (v2 
style)
* Should implement the {{SolrRequestHandler}} interface

The configuration looks as follows in {{solr.xml}} same as the configuration in 
{{solrconfig.xml}}

{code:xml}

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org