[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: (was: LUCENE-7493-Pass.patch)

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: LUCENE-7493-Pass.patch

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch, LUCENE-7493-Pass.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: (was: LUCENE-7493-Pass-V.20.patch)

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: (was: LUCENE-7493-Pass-TestCase.patch)

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: LUCENE-7493-Fail.patch

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: (was: LUCENE-7493-Fail-TestCase.patch)

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7493) Support of TotalHitCountCollector for FacetCollector.search api if numdocs passed as zero.

[Mahesh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahesh updated LUCENE-7493:
---
Attachment: (was: LUCENE-7493-Fail-V.20.patch)

> Support of TotalHitCountCollector for FacetCollector.search api if numdocs 
> passed as zero.
> --
>
> Key: LUCENE-7493
> URL: https://issues.apache.org/jira/browse/LUCENE-7493
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Mahesh
> Attachments: LUCENE-7493-Fail.patch
>
>
> Hi, 
> I want to do drill down search using FacetCollection below is the code 
> FacetsCollector facetCollector = new FacetsCollector();
> TopDocs topDocs = FacetsCollector.search(st.searcher, filterQuery, limit, 
> facetCollector);
> I just want facet information so I pass limit as zero but I get error 
> "numHits must be > 0; please use TotalHitCountCollector if you just need the 
> total hit count".
> For FacetCollector there is no way to initialize 'TotalHitCountCollector'. 
> Internally it always create either 'TopFieldCollector' or 
> 'TopScoreDocCollector' which does not allow limit as 0. 
> So if limit should be zero then there should be a way that 
> 'TotalHitCountCollector' should be initialized. 
> Better way would be to provide an api which takes query and collector as 
> inputs just like 'drillSideways.search(filterQuery, totalHitCountCollector)'.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Linux (64bit/jdk1.8.0_102) - Build # 1976 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Linux/1976/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ObjectTracker found 5 object(s) that were not released!!! 
[MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor, 
SolrCore, MockDirectoryWrapper] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:328)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:635)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:843)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:769)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:842)  at 
org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:66)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:667)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:843)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:769)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:842)  at 
org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at org.apache.solr.core.SolrCore.(SolrCore.java:792)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:769)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:842)  at 
org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at org.apache.solr.core.SolrCore.(SolrCore.java:933)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:769)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:842)  at 
org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:347)
  at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:430)  
at org.apache.solr.core.SolrCore.(SolrCore.java:837)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:769)  at 
org.apache.solr.core.CoreContainer.create(CoreContainer.java:842)  at 
org.apache.solr.core.CoreContainer.lambda$load$0(CoreContainer.java:498)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 

Re: lucene-solr:master: SOLR-9614: fixing TestSolrCloudWithKerberosAlt

[Mikhail Khludnev]

If we do that, how the following will be possible:

org.apache.solr.client.solrj.impl.SolrHttpClientContextBuilder.createContext(Object)

 public HttpClientContext createContext(Object userToken) {
HttpClientContext context = new HttpClientContext();

On Tue, Oct 18, 2016 at 2:48 AM, Chris Hostetter 
wrote:

>
> Alan & Mikhail...
>
> Should we add HttpClientContext's constructor to forbidden APIs list and
> force internal code to use the helper APIs to prevent (test) bugs like
> this in the future?
>
>
>
> : Date: Mon, 10 Oct 2016 09:22:51 + (UTC)
> : From: m...@apache.org
> : Reply-To: dev@lucene.apache.org
> : To: comm...@lucene.apache.org
> : Subject: lucene-solr:master: SOLR-9614: fixing
> TestSolrCloudWithKerberosAlt
> :
> : Repository: lucene-solr
> : Updated Branches:
> :   refs/heads/master d8bb56d4a -> 9fea5129d
> :
> :
> : SOLR-9614: fixing TestSolrCloudWithKerberosAlt
> :
> : Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
> : Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/
> 9fea5129
> : Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9fea5129
> : Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9fea5129
> :
> : Branch: refs/heads/master
> : Commit: 9fea5129d3eaef7cdc8086271677fc807ca1c020
> : Parents: d8bb56d
> : Author: Mikhail Khludnev 
> : Authored: Mon Oct 10 12:18:54 2016 +0300
> : Committer: Mikhail Khludnev 
> : Committed: Mon Oct 10 12:20:58 2016 +0300
> :
> : --
> :  .../java/org/apache/solr/client/solrj/impl/HttpClientUtil.java| 3
> +--
> :  1 file changed, 1 insertion(+), 2 deletions(-)
> : --
> :
> :
> : http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/
> 9fea5129/solr/solrj/src/java/org/apache/solr/client/solrj/
> impl/HttpClientUtil.java
> : --
> : diff --git 
> a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java
> b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/
> HttpClientUtil.java
> : index ed285d1..d4dea17 100644
> : --- a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/
> HttpClientUtil.java
> : +++ b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/
> HttpClientUtil.java
> : @@ -406,9 +406,8 @@ public class HttpClientUtil {
> : * in connection pools if client authentication is enabled.
> : */
> :public static HttpClientContext createNewHttpClientRequestContext() {
> : -HttpClientContext context = new HttpClientContext();
> : +HttpClientContext context = httpClientRequestContextBuilde
> r.createContext(HttpSolrClient.cacheKey);
> :
> : -context.setUserToken(HttpSolrClient.cacheKey);
> :  return context;
> :}
> :
> :
> :
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
Sincerely yours
Mikhail Khludnev

[jira] [Comment Edited] (SOLR-9599) DocValues performance regression with new iterator API

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584223#comment-15584223
 ] 

Yonik Seeley edited comment on SOLR-9599 at 10/18/16 3:18 AM:
--

Another docvalues faceting test, this time including the current lucene/solr 
code +  lucene70 codec (as of 10/17) 
This test used 10M documents and single valued string fields with 20% of the 
values missing (i.e. 80% of docs have a value for any given field).  4 
concurrent request threads were used with a 4 core CPU.
Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

This is a table of new_time/old_time, with old_time being an old docvalues 
index with old code (as of 9/09) before the docvalues iterator cutover:
||field cardinality||9/09 code with 9/09 index||10/17 code with 9/09 index|| 
10/17 code with 10/17 index||
| 10 | 1.00 | 1.39 | 1.41 |
| 100 | 1.00 | 1.38 | 1.46 |
| 1000 | 1.00 | 1.39 | 1.42 |
| 1 | 1.00 | 1.35 | 1.45 |

So it looks like we're currently over 40% slower in general for faceting on 
single valued docvalue fields that have some values missing.



was (Author: ysee...@gmail.com):
Another docvalues faceting test, this time including the current lucene/solr 
code +  lucene70 codec (as of 10/17) 
This test used 10M documents and single valued string fields with 20% of the 
values missing (i.e. 80% of docs have a value for any given field).
Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

This is a table of new_time/old_time, with old_time being an old docvalues 
index with old code (as of 9/09) before the docvalues iterator cutover:
||field cardinality||9/09 code with 9/09 index||10/17 code with 9/09 index|| 
10/17 code with 10/17 index||
| 10 | 1.00 | 1.39 | 1.41 |
| 100 | 1.00 | 1.38 | 1.46 |
| 1000 | 1.00 | 1.39 | 1.42 |
| 1 | 1.00 | 1.35 | 1.45 |

So it looks like we're currently over 40% slower in general for faceting on 
single valued docvalue fields that have some values missing.


> DocValues performance regression with new iterator API
> --
>
> Key: SOLR-9599
> URL: https://issues.apache.org/jira/browse/SOLR-9599
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Yonik Seeley
> Fix For: master (7.0)
>
>
> I did a quick performance comparison of faceting indexed fields (i.e. 
> docvalues are not stored) using method=dv before and after the new docvalues 
> iterator went in (LUCENE-7407).
> 5M document index, 21 segments, single valued string fields w/ no missing 
> values.
> || field cardinality || new_time / old_time ||
> |10|2.01|
> |1000|2.02|
> |1|1.85|
> |10|1.56|
> |100|1.31|
> So unfortunately, often twice as slow.
> See followup messages for tests using real docvalues as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7407) Explore switching doc values to an iterator API

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584286#comment-15584286
 ] 

Yonik Seeley commented on LUCENE-7407:
--

It seems like this has had a rather large impact on what was supposed to be the 
common case: dense doc values.  See SOLR-9599 for some performance numbers, 
currently standing at 40% slower for faceting as of today, but also impacting 
all other docvalue uses also tested (sorting and function queries).
Perhaps we should have both a random access API as well as an iterator API?

What do people thing about the right path forward?


> Explore switching doc values to an iterator API
> ---
>
> Key: LUCENE-7407
> URL: https://issues.apache.org/jira/browse/LUCENE-7407
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>  Labels: docValues
> Fix For: master (7.0)
>
> Attachments: LUCENE-7407.patch
>
>
> I think it could be compelling if we restricted doc values to use an
> iterator API at read time, instead of the more general random access
> API we have today:
>   * It would make doc values disk usage more of a "you pay for what
> what you actually use", like postings, which is a compelling
> reduction for sparse usage.
>   * I think codecs could compress better and maybe speed up decoding
> of doc values, even in the non-sparse case, since the read-time
> API is more restrictive "forward only" instead of random access.
>   * We could remove {{getDocsWithField}} entirely, since that's
> implicit in the iteration, and the awkward "return 0 if the
> document didn't have this field" would go away.
>   * We can remove the annoying thread locals we must make today in
> {{CodecReader}}, and close the trappy "I accidentally shared a
> single XXXDocValues instance across threads", since an iterator is
> inherently "use once".
>   * We could maybe leverage the numerous optimizations we've done for
> postings over time, since the two problems ("iterate over doc ids
> and store something interesting for each") are very similar.
> This idea has come up many in the past, e.g. LUCENE-7253 is a recent
> example, and very early iterations of doc values started with exactly
> this ;)
> However, it's a truly enormous change, likely 7.0 only.  Or maybe we
> could have the new iterator APIs also ported to 6.x side by side with
> the deprecate existing random-access APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9628) Trie fields have unset lastDocId

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584269#comment-15584269
 ] 

ASF subversion and git services commented on SOLR-9628:
---

Commit 19c7f8eacfccbe751279e6852fe6ec856403e658 in lucene-solr's branch 
refs/heads/master from [~anshum]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=19c7f8e ]

SOLR-9628: Set lastDocId in Trie fields


> Trie fields have unset lastDocId
> 
>
> Key: SOLR-9628
> URL: https://issues.apache.org/jira/browse/SOLR-9628
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Anshum Gupta
>Assignee: Anshum Gupta
> Attachments: SOLR-9628.patch, SOLR-9628.patch
>
>
> LUCENE-7407 switched doc values usage to an iterator API, introducing a 
> lastDocId to track in TrieLongField, TrieIntField, and TrieDoubleField. This 
> is never set. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-9599) DocValues performance regression with new iterator API

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584223#comment-15584223
 ] 

Yonik Seeley edited comment on SOLR-9599 at 10/18/16 2:59 AM:
--

Another docvalues faceting test, this time including the current lucene/solr 
code +  lucene70 codec (as of 10/17) 
This test used 10M documents and single valued string fields with 20% of the 
values missing (i.e. 80% of docs have a value for any given field).
Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

This is a table of new_time/old_time, with old_time being an old docvalues 
index with old code (as of 9/09) before the docvalues iterator cutover:
||field cardinality||9/09 code with 9/09 index||10/17 code with 9/09 index|| 
10/17 code with 10/17 index||
| 10 | 1.00 | 1.39 | 1.41 |
| 100 | 1.00 | 1.38 | 1.46 |
| 1000 | 1.00 | 1.39 | 1.42 |
| 1 | 1.00 | 1.35 | 1.45 |

So it looks like we're currently over 40% slower in general for faceting on 
single valued docvalue fields that have some values missing.



was (Author: ysee...@gmail.com):
Another docvalues faceting test, this time including the current lucene/solr 
code +  lucene70 codec (as of 10/17) 
This test used 10M documents and single valued string fields with 20% of the 
values missing (i.e. 80% of docs have a value for any given field).
Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

This is a table of new_time/old_time, with old_time being an old docvalues 
index with old code (as of 9/09) before the docvalues iterator cutover:
||field cardinality||10/17 code with 9/09 index|| 10/17 code with 10/17 index||
| 10 | 1.39 | 1.41 |
| 100 | 1.38 | 1.46 |
| 1000 | 1.39 | 1.42 |
| 1 | 1.35 | 1.45 |

So it looks like we're over 40% slower in general for faceting on single valued 
docvalue fields that have some values missing.


> DocValues performance regression with new iterator API
> --
>
> Key: SOLR-9599
> URL: https://issues.apache.org/jira/browse/SOLR-9599
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Yonik Seeley
> Fix For: master (7.0)
>
>
> I did a quick performance comparison of faceting indexed fields (i.e. 
> docvalues are not stored) using method=dv before and after the new docvalues 
> iterator went in (LUCENE-7407).
> 5M document index, 21 segments, single valued string fields w/ no missing 
> values.
> || field cardinality || new_time / old_time ||
> |10|2.01|
> |1000|2.02|
> |1|1.85|
> |10|1.56|
> |100|1.31|
> So unfortunately, often twice as slow.
> See followup messages for tests using real docvalues as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9599) DocValues performance regression with new iterator API

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-9599:
---
Description: 
I did a quick performance comparison of faceting indexed fields (i.e. docvalues 
are not stored) using method=dv before and after the new docvalues iterator 
went in (LUCENE-7407).

5M document index, 21 segments, single valued string fields w/ no missing 
values.

|| field cardinality || new_time / old_time ||
|10|2.01|
|1000|2.02|
|1|1.85|
|10|1.56|
|100|1.31|

So unfortunately, often twice as slow.

See followup messages for tests using real docvalues as well.

  was:
I did a quick performance comparison of faceting indexed fields (i.e. docvalues 
are not stored) using method=dv before and after the new docvalues iterator 
went in (LUCENE-7407).

5M document index, 21 segments, single valued string fields w/ no missing 
values.

|| field cardinality || new_time / old_time ||
|10|2.01|
|1000|2.02|
|1|1.85|
|10|1.56|
|100|1.31|

So unfortunately, often twice as slow.


> DocValues performance regression with new iterator API
> --
>
> Key: SOLR-9599
> URL: https://issues.apache.org/jira/browse/SOLR-9599
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Yonik Seeley
> Fix For: master (7.0)
>
>
> I did a quick performance comparison of faceting indexed fields (i.e. 
> docvalues are not stored) using method=dv before and after the new docvalues 
> iterator went in (LUCENE-7407).
> 5M document index, 21 segments, single valued string fields w/ no missing 
> values.
> || field cardinality || new_time / old_time ||
> |10|2.01|
> |1000|2.02|
> |1|1.85|
> |10|1.56|
> |100|1.31|
> So unfortunately, often twice as slow.
> See followup messages for tests using real docvalues as well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6188 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6188/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.lucene.store.TestSimpleFSDirectory

Error Message:
Could not remove the following files (in the order of attempts):
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001\tempDir-005:
 java.nio.file.AccessDeniedException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001\tempDir-005

C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001
 

Stack Trace:
java.io.IOException: Could not remove the following files (in the order of 
attempts):
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001\tempDir-005:
 java.nio.file.AccessDeniedException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001\tempDir-005
   
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001:
 java.nio.file.DirectoryNotEmptyException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\lucene\build\core\test\J1\temp\lucene.store.TestSimpleFSDirectory_4D86F1E019BCDC1A-001

at __randomizedtesting.SeedInfo.seed([4D86F1E019BCDC1A]:0)
at org.apache.lucene.util.IOUtils.rm(IOUtils.java:323)
at 
org.apache.lucene.util.TestRuleTemporaryFilesCleanup.afterAlways(TestRuleTemporaryFilesCleanup.java:216)
at 
com.carrotsearch.randomizedtesting.rules.TestRuleAdapter$1.afterAlways(TestRuleAdapter.java:31)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:43)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([EF326B30FE0BA291:878D5E1A2E91B07D]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 

[jira] [Updated] (SOLR-9599) DocValues performance regression with new iterator API

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-9599:
---
Summary: DocValues performance regression with new iterator API  (was: 
Facet performance regression using fieldcache and new DV iterator API)

> DocValues performance regression with new iterator API
> --
>
> Key: SOLR-9599
> URL: https://issues.apache.org/jira/browse/SOLR-9599
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Yonik Seeley
> Fix For: master (7.0)
>
>
> I did a quick performance comparison of faceting indexed fields (i.e. 
> docvalues are not stored) using method=dv before and after the new docvalues 
> iterator went in (LUCENE-7407).
> 5M document index, 21 segments, single valued string fields w/ no missing 
> values.
> || field cardinality || new_time / old_time ||
> |10|2.01|
> |1000|2.02|
> |1|1.85|
> |10|1.56|
> |100|1.31|
> So unfortunately, often twice as slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9599) Facet performance regression using fieldcache and new DV iterator API

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584223#comment-15584223
 ] 

Yonik Seeley commented on SOLR-9599:


Another docvalues faceting test, this time including the current lucene/solr 
code +  lucene70 codec (as of 10/17) 
This test used 10M documents and single valued string fields with 20% of the 
values missing (i.e. 80% of docs have a value for any given field).
Note that the 9/19 index has 24 segments and the 10/17 index has 23 segments.

This is a table of new_time/old_time, with old_time being an old docvalues 
index with old code (as of 9/09) before the docvalues iterator cutover:
||field cardinality||10/17 code with 9/09 index|| 10/17 code with 10/17 index||
| 10 | 1.39 | 1.41 |
| 100 | 1.38 | 1.46 |
| 1000 | 1.39 | 1.42 |
| 1 | 1.35 | 1.45 |

So it looks like we're over 40% slower in general for faceting on single valued 
docvalue fields that have some values missing.


> Facet performance regression using fieldcache and new DV iterator API
> -
>
> Key: SOLR-9599
> URL: https://issues.apache.org/jira/browse/SOLR-9599
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: master (7.0)
>Reporter: Yonik Seeley
> Fix For: master (7.0)
>
>
> I did a quick performance comparison of faceting indexed fields (i.e. 
> docvalues are not stored) using method=dv before and after the new docvalues 
> iterator went in (LUCENE-7407).
> 5M document index, 21 segments, single valued string fields w/ no missing 
> values.
> || field cardinality || new_time / old_time ||
> |10|2.01|
> |1000|2.02|
> |1|1.85|
> |10|1.56|
> |100|1.31|
> So unfortunately, often twice as slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Class#getResource returns null in JDK9 b140 if security manager is enabled (was: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!)

[Wang Weijun]

I do some investigation, looks like a method in 
randomizedtesting-runner-2.3.4.jar is trying to call a method in 
morfologik-polish-2.1.0.jar and this 2nd method uses getResource() to read 
something inside this 2nd jar.

This will fail because the 1st jar's ProtectionDomain is not granted the 
permission to read the 2nd jar. And yes, this means the 2nd jar should call 
getResource() in a doPrivileged block. Otherwise you need to grant that 
permission to every caller on the stack. This is not a bug.

This can be demonstrated with a simple example:

class T5 {
public static void main(String[] args) throws Exception {
go(args[0]);
}
public static void go(String arg) throws Exception {
System.out.println(new T5().getClass().getResource(arg));
}
}

class T6 {
public static void main(String[] args) throws Exception {
T5.go(args[0]);
}
}

Now you pack T5.class into 5.jar and T6.class into 6.jar.

5.jar is automatically granted the permission to read itself:

$ java -Djava.security.manager -cp 5.jar:6.jar T5 T5.class
jar:file:/Volumes/ServerHD/old/work/space/nb/build/classes/5.jar!/T5.class

But not 6.jar:

$ java -Djava.security.manager -cp 5.jar:6.jar T6 T5.class
null

Thanks
Max


> On Oct 18, 2016, at 1:44 AM, Uwe Schindler  wrote:
> 
> Hi,
> 
> we already had off-list contact, initiated by Rory O'Donnel - thanks!
> 
> The issue was indeed caused by symlinks. The issue here: The Jenkins server 
> where the tests are running had a home directory that was symlinked somewhere 
> else. All file paths during tests runs and also JAR files had the "correct" 
> (canonical path). But the homedir was defined with the alternate, symlinked 
> "old" path in /etc/passwd. Effect was that the test's policy file referring 
> to the IVY cache in ~/.ivy/cache for loading JAR files used the path 
> extracted from ${user.home}. Of course this broke.
> 
> I am sure this will hit many people, so I have some suggestions how to solve 
> this: In short, when parsing policy files and FilePermissions inside, just 
> "expand" the symlink to be canonic and add *both* (the symlink and the 
> canonic path) as 2 separate FilePermissions to the collection. This spares 
> the runtime check on every file access but still catches all "known" paths.
> 
> In addition, there is also a "bug" in the security permissions system that 
> made the above extra permission needed. We have some third party JARs, that 
> use Class#getResource() to load their own resources. But as getResource does 
> not document any security exceptions or other implications, almost all code 
> out there does not wrap with doPrivileged(). This very old bug required the 
> extra permission to the lib/ folder. The workaround just broke.
> 
> Here is what I wrote in the private discussion:
> 
> --snip--
>> Yes, this is where the problem is.
>> 
>> So it looks like the permission is granted in a policy file instead of being
>> granted by the class loader itself. In this case, the path of the permission
>> must match how you access that file.
> 
> Yes. I think the problem is that the 3rd party JAR file does not have a 
> doPrivileged block around the getResource/openStream part, so ist running 
> with the permissions of the calling code (a different JAR file - the test 
> runner). IMHO, this is one of the really strange things of the security model 
> in Java and most people do it wrong. Especially it is not clear from 
> Class#getResource that this can be affected by any security policy! It does 
> not even throw a declared SecurityException (because it is swallowed).
> 
> We have the extra path in our policy file for exactly that reason (to work 
> around issues in 3rd party JARs) that don't wrap this into doPrivileged!
> 
>> I'll think about your suggestion. However, can you guarantee the code always
>> accesses the file using the canonicalized path? For example, suppose the
>> actual file is /x, both /a and /b symlink to it, and you grant the 
>> permission on
>> /a in a policy file. Even if I canonicalize /a to /x and grant permissions on
>> both /a and /x, you will still see a failure if the code access /b.
> 
> I am coming from the full text search engine people. I see the issue that you 
> have with getting the canonical name on every file access (this slows down!). 
> The approach the full text people use is to make "synonyms" of terms and 
> index all of them. Somebody searching for the term will find it under any 
> name. To be ported over to your issue: Instead of doing the canonicalized 
> check on every access, just put *both* known file names into the "search 
> index" (in your case policy file). Means: When parsing the policy file, 
> create 2 file permissions: One as given in the policy and an additional one 
> with the canonical name. This does not solve all problems, but helps around 
> issues like the one we encountered.
> 
> I changed the setup of the Jenkins machine that hit this issue 

[jira] [Updated] (SOLR-9655) Add keyPhrases Streaming Expression

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9655:
-
Description: 
Following the work on the SOLR-9258, we can now perform parallel text analysis 
with Streaming Expressions. The first text analysis Streaming Expression is the 
classify() function. This ticket adds another text analysis function: 
keyPhrases().

The keyPhrases expression extracts the key phrases from a text field and 
attaches them to the tuples. The tuples can then be sent to another Solr Cloud 
collection for indexing.

  was:
Following the work on the SOLR-9258, we can now perform parallel text analysis 
with Streaming Expression. The first text analysis Streaming Expression is the 
classify() function. This ticket adds another text analysis function: 
keyPhrases().

The keyPhrases expression extracts the key phrases from a text field and 
attaches them to the tuples. The tuples can then be sent to another Solr Cloud 
collection for indexing.


> Add keyPhrases Streaming Expression
> ---
>
> Key: SOLR-9655
> URL: https://issues.apache.org/jira/browse/SOLR-9655
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> Following the work on the SOLR-9258, we can now perform parallel text 
> analysis with Streaming Expressions. The first text analysis Streaming 
> Expression is the classify() function. This ticket adds another text analysis 
> function: keyPhrases().
> The keyPhrases expression extracts the key phrases from a text field and 
> attaches them to the tuples. The tuples can then be sent to another Solr 
> Cloud collection for indexing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9655) Add keyPhrases Streaming Expression

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9655:
-
Summary: Add keyPhrases Streaming Expression  (was: Add keyPhrases 
Streaming Expressions)

> Add keyPhrases Streaming Expression
> ---
>
> Key: SOLR-9655
> URL: https://issues.apache.org/jira/browse/SOLR-9655
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> Following the work on the SOLR-9258, we can now perform parallel text 
> analysis with Streaming Expression. The first text analysis Streaming 
> Expression is the classify() function. This ticket adds another text analysis 
> function: keyPhrases().
> The keyPhrases expression extracts the key phrases from a text field and 
> attaches them to the tuples. The tuples can then be sent to another Solr 
> Cloud collection for indexing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9655) Add keyPhrases Streaming Expressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9655:
-
Summary: Add keyPhrases Streaming Expressions  (was: Add keyPhrase 
Streaming Expressions)

> Add keyPhrases Streaming Expressions
> 
>
> Key: SOLR-9655
> URL: https://issues.apache.org/jira/browse/SOLR-9655
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> Following the work on the SOLR-9258, we can now perform parallel text 
> analysis with Streaming Expression. The first text analysis Streaming 
> Expression is the classify() function. This ticket adds another text analysis 
> function: keyPhrases().
> The keyPhrases expression extracts the key phrases from a text field and 
> attaches them to the tuples. The tuples can then be sent to another Solr 
> Cloud collection for indexing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9655) Add keyPhrase Streaming Expressions

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9655:
-
Summary: Add keyPhrase Streaming Expressions  (was: Add keyPhrase Streaming 
Expression)

> Add keyPhrase Streaming Expressions
> ---
>
> Key: SOLR-9655
> URL: https://issues.apache.org/jira/browse/SOLR-9655
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>
> Following the work on the SOLR-9258, we can now perform parallel text 
> analysis with Streaming Expression. The first text analysis Streaming 
> Expression is the classify() function. This ticket adds another text analysis 
> function: keyPhrases().
> The keyPhrases expression extracts the key phrases from a text field and 
> attaches them to the tuples. The tuples can then be sent to another Solr 
> Cloud collection for indexing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9655) Add keyPhrase Streaming Expression

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-9655:


 Summary: Add keyPhrase Streaming Expression
 Key: SOLR-9655
 URL: https://issues.apache.org/jira/browse/SOLR-9655
 Project: Solr
  Issue Type: New Feature
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Joel Bernstein


Following the work on the SOLR-9258, we can now perform parallel text analysis 
with Streaming Expression. The first text analysis Streaming Expression is the 
classify() function. This ticket adds another text analysis function: 
keyPhrases().

The keyPhrases expression extracts the key phrases from a text field and 
attaches them to the tuples. The tuples can then be sent to another Solr Cloud 
collection for indexing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 458 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/458/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  org.apache.solr.cloud.DistribCursorPagingTest.test

Error Message:
java.util.concurrent.TimeoutException: Could not connect to ZooKeeper 
127.0.0.1:37976 within 3 ms

Stack Trace:
org.apache.solr.common.SolrException: java.util.concurrent.TimeoutException: 
Could not connect to ZooKeeper 127.0.0.1:37976 within 3 ms
at 
__randomizedtesting.SeedInfo.seed([3AB8044065F7511:8BFFBF9EA8A318E9]:0)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:182)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:116)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:111)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:98)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.printLayout(AbstractDistribZkTestBase.java:291)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.distribTearDown(AbstractFullDistribZkTestBase.java:1500)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:969)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.concurrent.TimeoutException: Could not connect to 
ZooKeeper 127.0.0.1:37976 within 3 ms
at 
org.apache.solr.common.cloud.ConnectionManager.waitForConnected(ConnectionManager.java:233)
at 
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:174)
... 37 more


FAILED:  

[JENKINS] Lucene-Solr-Tests-6.x - Build # 485 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-6.x/485/

1 tests failed.
FAILED:  
org.apache.solr.cloud.TestMiniSolrCloudCluster.testSegmentTerminateEarly

Error Message:
Error from server at http://127.0.0.1:43735/solr: Could not fully remove 
collection: testSegmentTerminateEarlyCollection

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:43735/solr: Could not fully remove collection: 
testSegmentTerminateEarlyCollection
at 
__randomizedtesting.SeedInfo.seed([46C18CD7BE96716:D4CADF60A7BBF37C]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:593)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:251)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:435)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1292)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1062)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1004)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.makeCollectionsRequest(MiniSolrCloudCluster.java:445)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.deleteCollection(MiniSolrCloudCluster.java:437)
at 
org.apache.solr.cloud.TestMiniSolrCloudCluster.testSegmentTerminateEarly(TestMiniSolrCloudCluster.java:448)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

Building a Solr cluster with Maven

[Greg Pendlebury]

Are there any developers with a current working maven build for a
downstream Solr installation? ie. Not a build for Solr itself, but a build
that brings in the core Solr server plus local plugins, third party plugins
etc?

I am in the process of updating one of our old builds (it builds both the
application and various shard instances) and have hit a stumbling block in
sourcing the dashboard static assets (everything under /webapp/web in
Solr's source).

Prior to the move away from being a webapp I could get them by exploding
the war from Maven Central.

In our very first foray into 5.x we had a local custom build to patch
SOLR-2649. We avoided solving this problem then by pushing the webapp into
our local Nexus as part of that build... but that wasn't a very good long
term choice.

So now I'm trying to work out the best long term approach to take here.
Ideas so far:

   1. Manually download the required zip and add it into our Nexus
   repository as a 3rd party artifact. Maven can source and extract anything
   it needs from here. This is where I'm currently leaning for simplicity, but
   the manual step required is annoying. It does have the advantage of causing
   a build failure straight away when a version upgrade occurs, prompting the
   developer to look into why.
   2. Move a copy of the static assets for the dashboard into our project
   and deploy them ourselves. This has the advantage of aligning our approach
   with the resources we already maintain in the project (like
   core.properties, schema.xml, solrconfig.xml, logging etc.). But I am
   worried that it is really fragile and developers will miss it during a
   version upgrade, resulting in the dashboard creeping out-of-date and
   (worse) introducing subtle bugs because of a version mismatch between the
   UI and the underlying server code.
   3. I'd like to think a long term approach would be for the core Solr
   build to ship a JAR (or any other assembly) to Maven Central like
   'solr-dashboard'... but I'm not sure how that aligns with the move away
   from Solr being considered a webapp. It seems a shame that all of the Java
   code ends up in Maven central, but the web layer dead-ends in the ant build.

I might be missing something really obvious and there is already a way to
do this. Is there some other distribution of the dashboard statics? Other
than the downloadable zip that is.

Ta,
Greg

Re: lucene-solr:master: SOLR-9614: fixing TestSolrCloudWithKerberosAlt

[Chris Hostetter]

Alan & Mikhail...

Should we add HttpClientContext's constructor to forbidden APIs list and 
force internal code to use the helper APIs to prevent (test) bugs like 
this in the future?



: Date: Mon, 10 Oct 2016 09:22:51 + (UTC)
: From: m...@apache.org
: Reply-To: dev@lucene.apache.org
: To: comm...@lucene.apache.org
: Subject: lucene-solr:master: SOLR-9614: fixing TestSolrCloudWithKerberosAlt
: 
: Repository: lucene-solr
: Updated Branches:
:   refs/heads/master d8bb56d4a -> 9fea5129d
: 
: 
: SOLR-9614: fixing TestSolrCloudWithKerberosAlt
: 
: Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
: Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/9fea5129
: Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/9fea5129
: Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/9fea5129
: 
: Branch: refs/heads/master
: Commit: 9fea5129d3eaef7cdc8086271677fc807ca1c020
: Parents: d8bb56d
: Author: Mikhail Khludnev 
: Authored: Mon Oct 10 12:18:54 2016 +0300
: Committer: Mikhail Khludnev 
: Committed: Mon Oct 10 12:20:58 2016 +0300
: 
: --
:  .../java/org/apache/solr/client/solrj/impl/HttpClientUtil.java| 3 +--
:  1 file changed, 1 insertion(+), 2 deletions(-)
: --
: 
: 
: 
http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/9fea5129/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java
: --
: diff --git 
a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java 
b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java
: index ed285d1..d4dea17 100644
: --- 
a/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java
: +++ 
b/solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpClientUtil.java
: @@ -406,9 +406,8 @@ public class HttpClientUtil {
: * in connection pools if client authentication is enabled.
: */
:public static HttpClientContext createNewHttpClientRequestContext() {
: -HttpClientContext context = new HttpClientContext();
: +HttpClientContext context = 
httpClientRequestContextBuilder.createContext(HttpSolrClient.cacheKey);
:  
: -context.setUserToken(HttpSolrClient.cacheKey);
:  return context;
:}
:
: 
: 

-Hoss
http://www.lucidworks.com/

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 914 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/914/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

2 tests failed.
FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([5EDFFF672537A67E:3660CA4DF5ADB492]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!

[Wang Weijun]

> On Oct 17, 2016, at 11:51 PM, Uwe Schindler  wrote:
> 
> Hi,
> 
>> One more thing:
>> 
>> You said "(although it should not be needed). Can you confirm that? I mean,
>> if you remove that line, does the test still passes with -
>> Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into
>> other reasons, since I did see in many other places file permission on
>> /home/jenkins//morfologik-polish-2.1.0.jar itself is granted.
> 
> See my other mail! It should not be needed, because Class#getResource does 
> not throw or document any Security implications. But in fact if you don't 
> wrap a doPrivileged around the call, it will also fail without our extra 
> permission. It was added to work around this issue with 3rd party JARs.

You mean this was observed before the new FilePermission change in 8164705?

If so, this means before the code change, you need FilePermissions on both 
/var/lib/.../- and /home/jenkins/.../thejar to make sure the jar is accessible. 
Here it seems the 2nd one should be enough but actually not, and the 1st is 
added to guarantee it. And after the code change, the 1st is useless and the 
test starts to fail.

I would consider adding a compatibility layer to make the 1st also useful, but 
at the same time, if we can fix the 2nd, that will be very helpful.

Thanks
Max

   
> 
> I just blew up with the symlink on home dir.
> 
> Uwe
> 
>> Thanks
>> Max
>> 
>> 
>>> On Oct 17, 2016, at 11:17 PM, Wang Weijun 
>> wrote:
>>> 
>>> Yes, this is where the problem is.
>>> 
>>> So it looks like the permission is granted in a policy file instead of being
>> granted by the class loader itself. In this case, the path of the permission
>> must match how you access that file.
>>> 
>>> I'll think about your suggestion. However, can you guarantee the code
>> always accesses the file using the canonicalized path? For example, suppose
>> the actual file is /x, both /a and /b symlink to it, and you grant the 
>> permission
>> on /a in a policy file. Even if I canonicalize /a to /x and grant 
>> permissions on
>> both /a and /x, you will still see a failure if the code access /b.
>>> 
>>> --Max
>>> 
 On Oct 17, 2016, at 11:03 PM, Uwe Schindler 
>> wrote:
 
 Hi,
 
 I attached the debugging logs ("all") for the working (conaonicalized) and
>> non-working case to this mail. Please keep in mind that around the
>> getResource() is no doPrivileged, so it looks like it uses privileges of 
>> outer JAR.
 
 I also checked, we have the following explicitely in our policy file, which
>> explicitly allows everything in the IVY cache (although it should not be
>> needed) - so it should really work anyways!!!
 
 permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-",
>> "read";
 
 But I think I know the issue - which is really stupid - but it’s a fact 
 affecting
>> mny people! Here is what the policy file parsing says above the
>> permission given before (please compare first line and second line):
 
 [junit4]   2>  Active Principals: []
 [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
 [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/var/lib/jenkins/.ivy2/cache/-" "read")
 [junit4]   2> policy:   granting ("java.io.FilePermission"
>> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
 
 Unfortunately, the user changed its home directory, the old one
>> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not
>> updated. So ${user.home} looks different. This is the reason why it did not
>> happen locally on my windows machine!
 
 IMHO: When reading the policy file, it should canonicalize all paths,
>> especially stuff like ${user.home}. At runtime when actually checking the
>> permissions it should not canonicalize for performance reasons. Not sure if
>> this is a bug, but this could hit lots of people. I know that other Jenkins
>> servers where Lucene runs their tests (not yet on Java 9) are setup in 
>> similar
>> ways!
 
 Uwe
 
 -
 Uwe Schindler
 H.-H.-Meier-Allee 63, D-28213 Bremen
 http://www.thetaphi.de
 eMail: u...@thetaphi.de
 
> -Original Message-
> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> Sent: Monday, October 17, 2016 4:03 PM
> To: dev@lucene.apache.org
> Cc: Dalibor Topic ; Balchandra Vaidya
> ; Muneer Kolarkunnu
> ; Rory O'Donnell
> 
> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140)
>> -
> Build # 18064 - Unstable!
> 
> OK, I'll try it tomorrow.
> 
> At the same time can you try -Djava.security.debug=all or -
> Djava.security.debug=scl and 

[jira] [Commented] (SOLR-9627) add QParser.getSortSpec, deprecate misleadingly named QParser.getSort

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583674#comment-15583674
 ] 

ASF subversion and git services commented on SOLR-9627:
---

Commit 30c9f4311a950d9387e5032b164a6a84ca1816c6 in lucene-solr's branch 
refs/heads/branch_6x from [~cpoerschke]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=30c9f43 ]

SOLR-9627: Add QParser.getSortSpec, deprecate misleadingly named 
QParser.getSort (Judith Silverman, Christine Poerschke)


> add QParser.getSortSpec, deprecate misleadingly named QParser.getSort
> -
>
> Key: SOLR-9627
> URL: https://issues.apache.org/jira/browse/SOLR-9627
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-9627.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-9641) Emit distributed tracing information from Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated SOLR-9641:

Comment: was deleted

(was: This is a fairly large patch that covers a lot of ground. A summary of 
the changes is:

* Introduce dependency on new version of HTrace. HTrace-3 (transitively 
required by Hadoop) cannot communicate with HTrace-4 (used by me here), but 
they are packaged/namespace separately and can exist in the same application. 
Hadoop will include HTrace-4 starting in version 2.8 and 3.0, I believe.
* Introduce new solr.xml config section, that passes all parameters to the 
HTrace configuration. If we decide to make the distributed tracing library we 
use pluggable, then I think we can spend some time to make the section a little 
more generic, but for now it is fine.
* Add trace object to CoreContainer. This seemed like a logical place for a 
singleton scoped to the entire Solr process, but I ran into issues when trying 
to add tracing to Overseer operations. This design decision may need to be 
revisited.
* Instrument SolrCore to handle all kinds of core related requests. This gets 
us most index handlers at once.
* Instrument a few of the executors we use to pass along trace ids for tracing 
over distribution requests.
* Instrument CollectionsHandler for create. Same pattern could be applied for 
other collection operations, but it may be a good idea to make this more 
generic later.
* Added a basic test. There is a start of a test for HDFS, but it's very much a 
work in progress.)

> Emit distributed tracing information from Solr
> --
>
> Key: SOLR-9641
> URL: https://issues.apache.org/jira/browse/SOLR-9641
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mike Drob
> Fix For: master (7.0)
>
> Attachments: SOLR-9641.patch
>
>
> While Solr already offers a few tools for exposing timing, this information 
> can be difficult to aggregate and analyze. By integrating distributed tracing 
> into Solr operations, we can gain new performance and behaviour insights.
> One such solution can be accomplished via Apache HTrace (incubating).
> (More rationale to follow.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9641) Emit distributed tracing information from Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated SOLR-9641:

Attachment: (was: SOLR-9641.patch)

> Emit distributed tracing information from Solr
> --
>
> Key: SOLR-9641
> URL: https://issues.apache.org/jira/browse/SOLR-9641
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mike Drob
> Fix For: master (7.0)
>
> Attachments: SOLR-9641.patch
>
>
> While Solr already offers a few tools for exposing timing, this information 
> can be difficult to aggregate and analyze. By integrating distributed tracing 
> into Solr operations, we can gain new performance and behaviour insights.
> One such solution can be accomplished via Apache HTrace (incubating).
> (More rationale to follow.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9641) Emit distributed tracing information from Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated SOLR-9641:

Attachment: SOLR-9641.patch

This is a fairly large patch that covers a lot of ground. A summary of the 
changes is:

* Introduce dependency on new version of HTrace. HTrace-3 (transitively 
required by Hadoop) cannot communicate with HTrace-4 (used by me here), but 
they are packaged/namespace separately and can exist in the same application. 
Hadoop will include HTrace-4 starting in version 2.8 and 3.0, I believe.
* Introduce new solr.xml config section, that passes all parameters to the 
HTrace configuration. If we decide to make the distributed tracing library we 
use pluggable, then I think we can spend some time to make the section a little 
more generic, but for now it is fine.
* Add trace object to CoreContainer. This seemed like a logical place for a 
singleton scoped to the entire Solr process, but I ran into issues when trying 
to add tracing to Overseer operations. This design decision may need to be 
revisited.
* Instrument SolrCore to handle all kinds of core related requests. This gets 
us most index handlers at once.
* Instrument a few of the executors we use to pass along trace ids for tracing 
over distribution requests.
* Instrument CollectionsHandler for create. Same pattern could be applied for 
other collection operations, but it may be a good idea to make this more 
generic later.
* Added a basic test. There is a start of a test for HDFS, but it's very much a 
work in progress.

> Emit distributed tracing information from Solr
> --
>
> Key: SOLR-9641
> URL: https://issues.apache.org/jira/browse/SOLR-9641
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mike Drob
> Fix For: master (7.0)
>
> Attachments: SOLR-9641.patch
>
>
> While Solr already offers a few tools for exposing timing, this information 
> can be difficult to aggregate and analyze. By integrating distributed tracing 
> into Solr operations, we can gain new performance and behaviour insights.
> One such solution can be accomplished via Apache HTrace (incubating).
> (More rationale to follow.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9641) Emit distributed tracing information from Solr

[Mike Drob (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Drob updated SOLR-9641:

Attachment: SOLR-9641.patch

This is a fairly large patch that covers a lot of ground. A summary of the 
changes is:

* Introduce dependency on new version of HTrace. HTrace-3 (transitively 
required by Hadoop) cannot communicate with HTrace-4 (used by me here), but 
they are packaged/namespace separately and can exist in the same application. 
Hadoop will include HTrace-4 starting in version 2.8 and 3.0, I believe.
* Introduce new solr.xml config section, that passes all parameters to the 
HTrace configuration. If we decide to make the distributed tracing library we 
use pluggable, then I think we can spend some time to make the section a little 
more generic, but for now it is fine.
* Add trace object to CoreContainer. This seemed like a logical place for a 
singleton scoped to the entire Solr process, but I ran into issues when trying 
to add tracing to Overseer operations. This design decision may need to be 
revisited.
* Instrument SolrCore to handle all kinds of core related requests. This gets 
us most index handlers at once.
* Instrument a few of the executors we use to pass along trace ids for tracing 
over distribution requests.
* Instrument CollectionsHandler for create. Same pattern could be applied for 
other collection operations, but it may be a good idea to make this more 
generic later.
* Added a basic test. There is a start of a test for HDFS, but it's very much a 
work in progress.

> Emit distributed tracing information from Solr
> --
>
> Key: SOLR-9641
> URL: https://issues.apache.org/jira/browse/SOLR-9641
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mike Drob
> Fix For: master (7.0)
>
> Attachments: SOLR-9641.patch
>
>
> While Solr already offers a few tools for exposing timing, this information 
> can be difficult to aggregate and analyze. By integrating distributed tracing 
> into Solr operations, we can gain new performance and behaviour insights.
> One such solution can be accomplished via Apache HTrace (incubating).
> (More rationale to follow.)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9627) add QParser.getSortSpec, deprecate misleadingly named QParser.getSort

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583632#comment-15583632
 ] 

ASF subversion and git services commented on SOLR-9627:
---

Commit c219525b1dfbe8ed0101d509754c9e5206f963de in lucene-solr's branch 
refs/heads/master from [~cpoerschke]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c219525 ]

SOLR-9627: Add QParser.getSortSpec, deprecate misleadingly named 
QParser.getSort (Judith Silverman, Christine Poerschke)


> add QParser.getSortSpec, deprecate misleadingly named QParser.getSort
> -
>
> Key: SOLR-9627
> URL: https://issues.apache.org/jira/browse/SOLR-9627
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-9627.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9654) add overrequest parameter to field faceting

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-9654:
---
Attachment: SOLR-9654.patch

Here's a draft patch w/o tests

> add overrequest parameter to field faceting
> ---
>
> Key: SOLR-9654
> URL: https://issues.apache.org/jira/browse/SOLR-9654
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Facet Module
>Reporter: Yonik Seeley
> Attachments: SOLR-9654.patch
>
>
> Add an "overrequest" parameter that can control the amount of overrequest 
> done for distributed search.  Among other things, this parameter will aid in 
> testing simple refinement cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9654) add overrequest parameter to field faceting

[Yonik Seeley (JIRA)]

Yonik Seeley created SOLR-9654:
--

 Summary: add overrequest parameter to field faceting
 Key: SOLR-9654
 URL: https://issues.apache.org/jira/browse/SOLR-9654
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Facet Module
Reporter: Yonik Seeley


Add an "overrequest" parameter that can control the amount of overrequest done 
for distributed search.  Among other things, this parameter will aid in testing 
simple refinement cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9280) make nodeName a configurable parameter in solr.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583388#comment-15583388
 ] 

ASF GitHub Bot commented on SOLR-9280:
--

Github user kelaban commented on the issue:

https://github.com/apache/lucene-solr/pull/50
  
Additionally work for SOLR-9281 is being done here 
(https://github.com/kelaban/lucene-solr/tree/jira/master/SOLR-9281) but will 
hold out on a PR as it relies on this branch


> make nodeName a configurable parameter in solr.xml
> --
>
> Key: SOLR-9280
> URL: https://issues.apache.org/jira/browse/SOLR-9280
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Keith Laban
>Assignee: Shalin Shekhar Mangar
>
> Originally node name is automatically generated based on 
> {{:_}}. Instead it should be configurable in solr.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #50: SOLR-9280 - make nodeName a configurable parameter in...

[kelaban]

Github user kelaban commented on the issue:

https://github.com/apache/lucene-solr/pull/50
  
Additionally work for SOLR-9281 is being done here 
(https://github.com/kelaban/lucene-solr/tree/jira/master/SOLR-9281) but will 
hold out on a PR as it relies on this branch


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583374#comment-15583374
 ] 

Christine Poerschke commented on SOLR-9512:
---

Thanks Shalin and Noble for analysing and summarising this.

The proposed cases 1-5 solution sounds good to me (though i have not actually 
looked at the code concerned to see what the implementation of that solution 
might look like).

Case 6: do i understand it right that we would keep failing the indexing 
requests but 'only' until eventually the client manages to reconnect to zk? I 
agree that asking a random solr instance for its latest cluster state could be 
problematic.


> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9512.patch
>
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9280) make nodeName a configurable parameter in solr.xml

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583342#comment-15583342
 ] 

ASF GitHub Bot commented on SOLR-9280:
--

Github user kelaban commented on the issue:

https://github.com/apache/lucene-solr/pull/50
  
Updated so the patch now applies cleanly. I've been having trouble getting 
tests to pass on vanilla master branch, so I cannot tell if this is causing any 
to fail. I've spot checked a few tests with this patch and they worked fine


> make nodeName a configurable parameter in solr.xml
> --
>
> Key: SOLR-9280
> URL: https://issues.apache.org/jira/browse/SOLR-9280
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Keith Laban
>Assignee: Shalin Shekhar Mangar
>
> Originally node name is automatically generated based on 
> {{:_}}. Instead it should be configurable in solr.xml



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr issue #50: SOLR-9280 - make nodeName a configurable parameter in...

[kelaban]

Github user kelaban commented on the issue:

https://github.com/apache/lucene-solr/pull/50
  
Updated so the patch now applies cleanly. I've been having trouble getting 
tests to pass on vanilla master branch, so I cannot tell if this is causing any 
to fail. I've spot checked a few tests with this patch and they worked fine


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8552) Unbalanced quotes in bin/solr when -D arguments are passed via the -a switch

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583297#comment-15583297
 ] 

Jan Høydahl commented on SOLR-8552:
---

Managed to reproduce with
{noformat}
bin/solr -e techproducts -a "-Dfoo -Dbar"
{noformat}

> Unbalanced quotes in bin/solr when -D arguments are passed via the -a switch
> 
>
> Key: SOLR-8552
> URL: https://issues.apache.org/jira/browse/SOLR-8552
> Project: Solr
>  Issue Type: Bug
>  Components: scripts and tools
>Reporter: Shalin Shekhar Mangar
>Priority: Minor
>
> This works:
> {code}
> bin/solr start -p 8983 -h localhost -m 2g -e schemaless 
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false
> {code}
> but this does not:
> {code}
> bin/solr start -p 8983 -h localhost -m 2g -e schemaless -a 
> "-Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port= 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false"
> {code}
> The output for the latter is:
> {code}
> Solr home directory 
> /home/shalin/temp/bench/solr/wiki-4k-schema/example/schemaless/solr already 
> exists.
> Starting up Solr on port 8983 using command:
> bin/solr start -p 8983 -s "example/schemaless/solr" -m 2g 
> -Dcom.sun.management.jmxremote.port= 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false" -a "-Dcom.sun.management.jmxremote"
> ERROR: Unbalanced quotes in bin/solr start -p 8983 -s 
> "example/schemaless/solr" -m 2g -Dcom.sun.management.jmxremote.port= 
> -Dcom.sun.management.jmxremote.authenticate=false 
> -Dcom.sun.management.jmxremote.ssl=false" -a "-Dcom.sun.management.jmxremote"
> {code}
> I know bin/solr supports direct pass through of -D properties but it should 
> still work with -a option because that is how many people would have 
> configured -D properties before support for the pass-through was added.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6914) DecimalDigitFilter skips characters in some cases (supplemental?)

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583280#comment-15583280
 ] 

Hoss Man commented on LUCENE-6914:
--

go for it [~jpountz] ... like i said before...

bq. Since i'm way out of my depth here i don't intend on commiting ... Anybody 
who understands this and thinks my patch looks good is welcome to run with it 
... no need to wait for me.

> DecimalDigitFilter skips characters in some cases (supplemental?)
> -
>
> Key: LUCENE-6914
> URL: https://issues.apache.org/jira/browse/LUCENE-6914
> Project: Lucene - Core
>  Issue Type: Bug
>Affects Versions: 5.4
>Reporter: Hoss Man
> Attachments: LUCENE-6914.patch, LUCENE-6914.patch, LUCENE-6914.patch
>
>
> Found this while writing up the solr ref guide for DecimalDigitFilter. 
> With input like "ퟙퟡퟠퟜ" ("Double Struck" 1984) the filter produces "1ퟡ8ퟜ" (1, 
> double struck 9, 8, double struck 4)  add some non-decimal characters in 
> between the digits (ie: "ퟙxퟡxퟠxퟜ") and you get the expected output 
> ("1x9x8x4").  This doesn't affect all decimal characters though, as evident 
> by the existing test cases.
> Perhaps this is an off by one bug in the "if the original was supplementary, 
> shrink the string" code path?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9255) Start Script Basic Authentication

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583271#comment-15583271
 ] 

Martin Löper commented on SOLR-9255:


Yes, I agree on changing the affected version to 7.0 (master)!

> Start Script Basic Authentication
> -
>
> Key: SOLR-9255
> URL: https://issues.apache.org/jira/browse/SOLR-9255
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Affects Versions: master (7.0)
>Reporter: Martin Löper
>Assignee: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-9255.patch, SOLR-9255.patch
>
>
> I configured SSL and BasicAuthentication with Rule-Based-Authorization.
> I noticed that since the latest changes from 6.0.1 to 6.1.0 I cannot pass the 
> Basic Authentication Credentials to the Solr Start Script anymore. For the 
> previous release I did this via the bin/solr.in.sh shellscript.
> What has happened with the SOLR_AUTHENTICATION_CLIENT_CONFIGURER and 
> SOLR_AUTHENTICATION_OPTS parameters? Are they still in use or is there a new 
> way to pass basic auth credentials on the command-line?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9255) Start Script Basic Authentication

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-9255:
--
Attachment: SOLR-9255.patch

New patch which is tested on Windows.

I might have added the {{%AUTHC_OPTS%}} a few places too much (20 places), 
anyone familiar with which commands really need it?

Still have not tested setting $SOLR_AUTHENTICATION_CLIENT_BUILDER variable, as 
I need an example of how to test it. Anyone?

> Start Script Basic Authentication
> -
>
> Key: SOLR-9255
> URL: https://issues.apache.org/jira/browse/SOLR-9255
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Affects Versions: master (7.0)
>Reporter: Martin Löper
>Assignee: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-9255.patch, SOLR-9255.patch
>
>
> I configured SSL and BasicAuthentication with Rule-Based-Authorization.
> I noticed that since the latest changes from 6.0.1 to 6.1.0 I cannot pass the 
> Basic Authentication Credentials to the Solr Start Script anymore. For the 
> previous release I did this via the bin/solr.in.sh shellscript.
> What has happened with the SOLR_AUTHENTICATION_CLIENT_CONFIGURER and 
> SOLR_AUTHENTICATION_OPTS parameters? Are they still in use or is there a new 
> way to pass basic auth credentials on the command-line?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_102) - Build # 18075 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18075/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.PeerSyncReplicationTest.test

Error Message:
expected:<152> but was:<138>

Stack Trace:
java.lang.AssertionError: expected:<152> but was:<138>
at 
__randomizedtesting.SeedInfo.seed([B34C2CDBFAC81D8C:3B18130154347074]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.bringUpDeadNodeAndEnsureNoReplication(PeerSyncReplicationTest.java:280)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.forceNodeFailureAndDoPeerSync(PeerSyncReplicationTest.java:244)
at 
org.apache.solr.cloud.PeerSyncReplicationTest.test(PeerSyncReplicationTest.java:130)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 

[jira] [Comment Edited] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583135#comment-15583135
 ] 

Jan Høydahl edited comment on SOLR-8370 at 10/17/16 7:01 PM:
-

New patch:
* Fixed javaScript error msg when fetching similarity for a partialState field
* Fixed display of warning message about incomplete info in partialState
!screenshot-5.png|width=500!
* Hidden empty {{FieldType}} label in partialState
* Hidden {{Load Term Info}} button in partialState
* Verified that {{author_s}} displays with full info once we index a document 
with {{author}} field, i.e. 
{noformat}
bin/post -c techproducts example/exampledocs/books.csv
{noformat}


was (Author: janhoy):
New patch:
* Fixed javaScript error msg when fetching similarity for a partialState field
* Fixed display of warning message about incomplete info in partialState
!screenshot-5.png!
* Hidden empty {{FieldType}} label in partialState
* Hidden {{Load Term Info}} button in partialState
* Verified that {{author_s}} displays with full info once we index a document 
with {{author}} field, i.e. 
{noformat}
bin/post -c techproducts example/exampledocs/books.csv
{noformat}

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png, 
> screenshot-5.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-8370:
--
Attachment: screenshot-5.png

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png, 
> screenshot-5.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-8370:
--
Attachment: SOLR-8370.patch

New patch:
* Fixed javaScript error msg when fetching similarity for a partialState field
* Fixed display of warning message about incomplete info in partialState
!screenshot-5.png!
* Hidden empty {{FieldType}} label in partialState
* Hidden {{Load Term Info}} button in partialState
* Verified that {{author_s}} displays with full info once we index a document 
with {{author}} field, i.e. 
{noformat}
bin/post -c techproducts example/exampledocs/books.csv
{noformat}

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> screenshot-1.png, screenshot-2.png, screenshot-3.png, screenshot-4.png, 
> screenshot-5.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6730) select?replicaAffinity=(node|host) and replicaAffinity.hostPriorities support

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-6730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583110#comment-15583110
 ] 

Christine Poerschke commented on SOLR-6730:
---

Noble and I discussed offline re: SOLR-6730 and SOLR-8146 overlaps and 
differences. I will try to summarise here as bullet points, [~noble.paul] 
please add or correct if i missed or misunderstood something.

* The use case and motivation for the {{select?replicaAffinity=(node|host)}} 
part of SOLR-6730 was to reduce the number of JVMs hit by a given search since 
the more JVMs are hit, the higher the chance of hitting a garbage collection 
pause in one of many JVMs.

* The use case and motivation for the {{replicaAffinity.hostPriorities=...}} 
part of SOLR-6730 was to preferentially direct requests from the same 
user/source to certain areas of the cloud.
** The implementation of the {{replicaAffinity.hostPriorities=...}} approach 
requires configuration somewhere i.e. a list of which hosts to prioritise.
** No matter where it is stored, maintaining configuration can be cumbersome as 
collections and hosts change over time.

* The objective of directing requests from the same user/source to certain 
areas of the cloud can be achieved without configuration, and the objective of 
reducing the number of JVMs hit by a search can pretty much be achieved that 
way also.
** Approach outline:
*** Two numeric parameters ('seed' and 'mod') are optionally added to each 
request.
*** The two parameters 'place' the requests within the cloud, e.g. for 
{{mod=9}} any seed between 0 and 8 would be valid and {{seed=6}} would 'place' 
the request with the 7th of 9 replicas, or more realistically the 3rd of 3 
replicas.
*** seed-plus-mod placement automatically adjusts when the number of replicas 
changes i.e. (seed=2,mod=6) would be 3rd-of-6 or 2nd-of-4 or 2nd-of-3 or 
1st-of-2 placement.
*** SOLR-6730 here would likely be abandoned in favour of the approach outlined.

* What is common to SOLR-6730 and SOLR-8146:
** optional parameters would support changing of the existing behaviour
** existing behaviour is maintained if the optional parameters are not supplied

* What is different between SOLR-6730 and SOLR-8146:
** point-of-use of the optional parameter is HttpShardHandler\[Factory\] for 
SOLR-6730
** point-of-use of the optional parameter is CloudSolrClient (and 
HttpShardHandler\[Factory\]?) for SOLR-8146

* Next steps:
1. SOLR-8332 to factor HttpShardHandler\[Factory\]'s url shuffling out into a 
ReplicaListTransformer class
2. creation of additional ReplicaListTransformer implementations corresponding 
to the approach outlined above

> select?replicaAffinity=(node|host) and replicaAffinity.hostPriorities support
> -
>
> Key: SOLR-6730
> URL: https://issues.apache.org/jira/browse/SOLR-6730
> Project: Solr
>  Issue Type: New Feature
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
>
> If no shards parameter is supplied with a select request then sub-requests 
> will go to a random selection of live solr nodes hosting shards for the 
> collection of interest. All sub-requests must complete before results can be 
> collated i.e. the slowest sub-request determines how fast the search 
> completes.
> Use of optional replicaAffinity can reduce the number of JVMs hit by a given 
> search (the more JVMs are hit, the higher the chance of hitting a garbage 
> collection pause in one of many JVMs). Preferentially directing requests to 
> certain areas of the cloud can also be useful for debugging or when some 
> replicas reside on 'faster' machines.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9417) Allow daemons to terminate when they finish iterating a topic

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9417:
-
Summary: Allow daemons to terminate when they finish iterating a topic  
(was: Allow daemons to terminate)

> Allow daemons to terminate when they finish iterating a topic
> -
>
> Key: SOLR-9417
> URL: https://issues.apache.org/jira/browse/SOLR-9417
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: 6.3
>
> Attachments: SOLR-9417.patch, SOLR-9417.patch
>
>
> The daemon expression currently runs until it's killed. This ticket will add 
> a new *terminate* parameter to the daemon expression that will allow the 
> daemon to shut itself down when it's finished processing a topic.
> There are a couple of small changes that need to be made to allow the daemon 
> to terminate on it's own:
> 1) The daemon will need to be passed the Map of all daemons in the /stream 
> handler. This will allow the DaemonStream to remove itself from the Map when 
> it terminates.
> 2) Logic needs to be added for the daemon to exit it's run loop if the topic 
> signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can 
> be used for this purpose. If sleepMillis is greater then 0 then this signals 
> a zero Tuple run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Mike (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583099#comment-15583099
 ] 

Mike commented on SOLR-9651:


Well, there's also SOLR-9653 which is specifically about deprecating EFFs. 
Might be a better place to discuss and then come back here if the decision is 
not to deprecate?

> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9417) Allow daemons to terminate

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9417:
-
Description: 
The daemon expression currently runs until it's killed. This ticket will add a 
new *terminate* parameter to the daemon expression that will allow the daemon 
to shut itself down when it's finished processing a topic.

There are a couple of small changes that need to be made to allow the daemon to 
terminate on it's own:

1) The daemon will need to be passed the Map of all daemons in the /stream 
handler. This will allow the DaemonStream to remove itself from the Map when it 
terminates.
2) Logic needs to be added for the daemon to exit it's run loop if the topic 
signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can 
be used for this purpose. If sleepMillis is greater then 0 then this signals a 
zero Tuple run.

  was:
The daemon expression currently runs until it's killed. This ticket will add a 
new *terminate* parameter to the daemon expression that will allow the daemon 
to shut itself down when it's finished processing a topic queue.

There are a couple of small changes that need to be made to allow the daemon to 
terminate on it's own:

1) The daemon will need to be passed the Map of all daemons in the /stream 
handler. This will allow the DaemonStream to remove itself from the Map when it 
terminates.
2) Logic needs to be added for the daemon to exit it's run loop if the topic 
signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can 
be used for this purpose. If sleepMillis is greater then 0 then this signals a 
zero Tuple run.


> Allow daemons to terminate
> --
>
> Key: SOLR-9417
> URL: https://issues.apache.org/jira/browse/SOLR-9417
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: 6.3
>
> Attachments: SOLR-9417.patch, SOLR-9417.patch
>
>
> The daemon expression currently runs until it's killed. This ticket will add 
> a new *terminate* parameter to the daemon expression that will allow the 
> daemon to shut itself down when it's finished processing a topic.
> There are a couple of small changes that need to be made to allow the daemon 
> to terminate on it's own:
> 1) The daemon will need to be passed the Map of all daemons in the /stream 
> handler. This will allow the DaemonStream to remove itself from the Map when 
> it terminates.
> 2) Logic needs to be added for the daemon to exit it's run loop if the topic 
> signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can 
> be used for this purpose. If sleepMillis is greater then 0 then this signals 
> a zero Tuple run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9417) Allow daemons to terminate

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-9417:
-
Attachment: SOLR-9417.patch

Added a very simple test case, which shows a daemon terminating when the 
underlying topic returns no Tuples. More to follow...

> Allow daemons to terminate
> --
>
> Key: SOLR-9417
> URL: https://issues.apache.org/jira/browse/SOLR-9417
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Fix For: 6.3
>
> Attachments: SOLR-9417.patch, SOLR-9417.patch
>
>
> The daemon expression currently runs until it's killed. This ticket will add 
> a new *terminate* parameter to the daemon expression that will allow the 
> daemon to shut itself down when it's finished processing a topic queue.
> There are a couple of small changes that need to be made to allow the daemon 
> to terminate on it's own:
> 1) The daemon will need to be passed the Map of all daemons in the /stream 
> handler. This will allow the DaemonStream to remove itself from the Map when 
> it terminates.
> 2) Logic needs to be added for the daemon to exit it's run loop if the topic 
> signals it had a zero Tuple run. The *sleepMillis* value in the EOF Tuple can 
> be used for this purpose. If sleepMillis is greater then 0 then this signals 
> a zero Tuple run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Keith Laban (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583091#comment-15583091
 ] 

Keith Laban commented on SOLR-9651:
---

I wrote an extension of EFF called RemoteFileField (SOLR-9617). The idea is 
that you can drop your external file field in an s3 bucket or some remote 
hosted place and then tell solr to suck it down and update the EFF. We could 
probably use the same approach to have it just do atomic updates to the 
documents instead of writing an external file. 

Maybe the title/description of this ticket should be updated to be a discussion 
around finding a better approach for EFF.

> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582392#comment-15582392
 ] 

Jan Høydahl edited comment on SOLR-8370 at 10/17/16 6:48 PM:
-

-Alexandre, would you care to test the JS bug with a plain trunk version and 
see if you can reproduce without this patch, and file a JIRA if necessary?- 
*Forget this*

I'm changing wording from Default -> Global
If the other issues are unrelated to this one, then are there more stuff to 
change for this JIRA?


was (Author: janhoy):
Alexandre, would you care to test the JS bug with a plain trunk version and see 
if you can reproduce without this patch, and file a JIRA if necessary?

I'm changing wording from Default -> Global
If the other issues are unrelated to this one, then are there more stuff to 
change for this JIRA?

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, screenshot-1.png, 
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582370#comment-15582370
 ] 

Jan Høydahl edited comment on SOLR-8370 at 10/17/16 6:48 PM:
-

Looking at author_s, it is broken from Luke. The endpoint 
http://localhost:8983/solr/techproducts/admin/luke does not contain this 
dynamic field (but it contains address_s), neither does while 
http://localhost:8983/solr/techproducts/admin/luke?show=schema but it is listed 
as a copyDest. 

I'm not sure if Luke was supposed to bring it back as a field, but tell us that 
it is a dynField, or whether {{mergeIndexAndSchemaData(index, schema.schema);}} 
method in schema.js was supposed to find out that by itself, as it is now, it 
is that method that populates {{author_s}} but in a partial:true state... It 
also has a line 
{code}
display.partialState = is.field && !!data.fields[name].partial;
{code}
-which seems to set a variabel but do nothing further with it. So there may be 
some work in progress here that was never finalised? This also smells like a 
new JIRA, not part of this one...-

*UPDATE*: partialState is attempted used in {{schema.html}} but due to a typo, 
it does not work. I will fix that too (use {{display.partialState}} instead of 
just {{partialState}}. That will display a warning box "Because your Index is 
empty, we do not have enough Information about this Field".


was (Author: janhoy):
Looking at author_s, it is broken from Luke. The endpoint 
http://localhost:8983/solr/techproducts/admin/luke does not contain this 
dynamic field (but it contains address_s), neither does while 
http://localhost:8983/solr/techproducts/admin/luke?show=schema but it is listed 
as a copyDest. 

I'm not sure if Luke was supposed to bring it back as a field, but tell us that 
it is a dynField, or whether {{mergeIndexAndSchemaData(index, schema.schema);}} 
method in schema.js was supposed to find out that by itself, as it is now, it 
is that method that populates {{author_s}} but in a partial:true state... It 
also has a line 
{code}
display.partialState = is.field && !!data.fields[name].partial;
{code}
which seems to set a variabel but do nothing further with it. So there may be 
some work in progress here that was never finalised? This also smells like a 
new JIRA, not part of this one...

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, screenshot-1.png, 
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Mike (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583073#comment-15583073
 ] 

Mike commented on SOLR-9651:


That could be a solution, but our EFF file is about 90MB...so even with long 
updates we'd still have to send a lot of queries (or one TRULY massive one!). 

I'm happy so long as I have a way to quickly update a single numeric field for 
X million documents. If doing it via HTTP is easiest, that's fine, provided 
performance is reasonable.

> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583037#comment-15583037
 ] 

Erick Erickson commented on SOLR-9651:
--

Good points. I wonder if the ability to send very long updates would serve? By 
that I mean instead of writing a file, a mechanism whereby I could send the 
gazillion updates to a DV field at once?

Mind you, I don't really have a lot of investment in deprecating EFF, mostly 
I'd like to be sure we discuss it and explore possibilities before embarking on 
a fix.



> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Keith Laban (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583026#comment-15583026
 ] 

Keith Laban commented on SOLR-9651:
---

I wonder if we can do something even more hacky (and efficient) like write the 
external file field as a doc value segment and pretend like its an unstored, 
doc value field. I'm with Mike in thinking that dropping a file is much easier 
than updating all your documents.

> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9651) Consider tracking modification time of external file fields for faster reloading

[Mike (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583004#comment-15583004
 ] 

Mike commented on SOLR-9651:


I haven't read the super long SOLR-5944 issue, but the one thing I'll say about 
external file fields that's wonderful is that they can be updated in a flash. 
For a database with millions of documents, I can update them all just by 
updating a single file. That's very powerful and it'd be a shame if EFF were 
deprecated and I had to send a gazillion update queries to Solr to do the same.

> Consider tracking modification time of external file fields for faster 
> reloading
> 
>
> Key: SOLR-9651
> URL: https://issues.apache.org/jira/browse/SOLR-9651
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 4.10.4
> Environment: Linux
>Reporter: Mike
>
> I have an index of about 4M legal documents that has pagerank boosting 
> configured as an external file field. The external file is about 100MB in 
> size and has one row per document in the index. Each row indicates the 
> pagerank score of a document. When we open new searchers, this file has to 
> get reloaded, and it creates a noticeable delay for our users -- takes 
> several seconds to reload. 
> An idea to fix this came up in [a recent discussion in the Solr mailing 
> list|https://www.mail-archive.com/solr-user@lucene.apache.org/msg125521.html]:
>  Could the file only be reloaded if it has changed on disk? In other words, 
> when new searchers are opened, could they check the modtime of the file, and 
> avoid reloading it if the file hasn't changed? 
> In our configuration, this would be a big improvement. We only change the 
> pagerank file once/week because computing it is intensive and new documents 
> don't tend to have a big impact. At the same time, because we're regularly 
> adding new documents, we do hundreds of commits per day, all of which have a 
> delay as the (largish) external file field is reloaded. 
> Is this a reasonable improvement to request? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9483) Add SolrJ support for the modify collection API

[Jacob Motley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582986#comment-15582986
 ] 

Jacob Motley commented on SOLR-9483:


Where is a good place to start to correctly add this method to 
CollectionAdminRequest? Are you asking for multiple "set" methods or something 
like a single method with the ability to change all of the attributes of the 
Collection it is called on?

> Add SolrJ support for the modify collection API
> ---
>
> Key: SOLR-9483
> URL: https://issues.apache.org/jira/browse/SOLR-9483
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: clients - java, SolrCloud, SolrJ
>Reporter: Shalin Shekhar Mangar
>  Labels: difficulty-easy, newdev
> Fix For: 6.3, master (7.0)
>
>
> SolrJ currently does not have a method corresponding to the modify collection 
> API. There should be a Modify class inside CollectionAdminRequest and a 
> simple method to change all parameters supported by the modify API.
> Link to modify API documentation: 
> https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-modifycoll



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582967#comment-15582967
 ] 

Hrishikesh Gadre commented on SOLR-9642:


[~yo...@apache.org] Ok cool!

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
> Fix For: 6.3
>
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582962#comment-15582962
 ] 

Yonik Seeley commented on SOLR-9642:


It was... it just hasn't shown up in this JIRA for some reason.
https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;a=commit;h=46aeb52588c6ecf298ee894a79fb162e4e3437fe

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
> Fix For: 6.3
>
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Hrishikesh Gadre (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582949#comment-15582949
 ] 

Hrishikesh Gadre commented on SOLR-9642:


[~yo...@apache.org] Thanks for taking a look! Shouldn't this be committed to 
trunk as well?

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
> Fix For: 6.3
>
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley resolved SOLR-9642.

   Resolution: Fixed
Fix Version/s: 6.3

Committed. Thanks!

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
> Fix For: 6.3
>
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582939#comment-15582939
 ] 

ASF subversion and git services commented on SOLR-9642:
---

Commit 06ae3eb2ec71f97b39a4ab0b7579cda0397d9a1b in lucene-solr's branch 
refs/heads/branch_6x from [~hgadre]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=06ae3eb ]

SOLR-9642: Refactor the snapshot cleanup mechanism to rely on Lucene

The current snapshot cleanup mechanism is based on reference counting
the index files shared between multiple segments. Since this mechanism
completely skips the Lucene APIs, it is not portable (e.g. it doesn't
work on 4.10.x version).

This patch provides an alternate implementation which relies exclusively
on Lucene IndexWriter (+ IndexDeletionPolicy) for cleanup.

mend


> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
> Fix For: 6.3
>
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7494) Explore making PointValues a per-field API like doc values

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582913#comment-15582913
 ] 

Michael McCandless commented on LUCENE-7494:


+1 to the patch, thanks [~jpountz]!

> Explore making PointValues a per-field API like doc values
> --
>
> Key: LUCENE-7494
> URL: https://issues.apache.org/jira/browse/LUCENE-7494
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7494.patch
>
>
> This is a follow-up to LUCENE-7491. Maybe we could simplify things a bit by 
> changing {{LeafReader.getPointValues()}} to 
> {{LeafReader.getPointValues(String fieldName)}} and removing all {{String 
> fieldName}} parameters from {{PointValues}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9566) Can we avoid doing recovery when collections are first created?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582907#comment-15582907
 ] 

ASF subversion and git services commented on SOLR-9566:
---

Commit 65f55802ee01b90a7f529de270d5d866a2282a40 in lucene-solr's branch 
refs/heads/master from [~romseygeek]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=65f5580 ]

SOLR-9566: Don't put replicas into recovery when collections are created


> Can we avoid doing recovery when collections are first created?
> ---
>
> Key: SOLR-9566
> URL: https://issues.apache.org/jira/browse/SOLR-9566
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9566.patch
>
>
> When a core starts up as part of a collection, and it's not a shard leader, 
> it goes into recovery, part of which involves a 7 second wait (set in 
> SOLR-7141) to ensure that updates being sent from the leader don't get missed 
> in between buffering and replication.
> This has the unfortunate side-effect of adding a 7-second pause to collection 
> creation whenever the replication factor is 2 or more, which slows down tests 
> massively - for example, DeleteReplicaTest takes about 54 seconds to execute 
> on my machine, 28 seconds of which is just pauses - over 50% of execution 
> time.  It's not actually possible to add documents to a collection before the 
> creation request has returned, so the recovery stage here isn't strictly 
> speaking necessary.  
> I think we could try adding a parameter to a CoreAdmin create request that 
> says the core is being created as part of a new collection, so it doesn't 
> need to try and recover from it's leader when it starts up.  Does this sound 
> sensible, or am I missing something here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9566) Can we avoid doing recovery when collections are first created?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582906#comment-15582906
 ] 

ASF subversion and git services commented on SOLR-9566:
---

Commit c9eb9e10e4168ae1fba6d122ab80aa1402f33fce in lucene-solr's branch 
refs/heads/branch_6x from [~romseygeek]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c9eb9e1 ]

SOLR-9566: Don't put replicas into recovery when collections are created


> Can we avoid doing recovery when collections are first created?
> ---
>
> Key: SOLR-9566
> URL: https://issues.apache.org/jira/browse/SOLR-9566
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9566.patch
>
>
> When a core starts up as part of a collection, and it's not a shard leader, 
> it goes into recovery, part of which involves a 7 second wait (set in 
> SOLR-7141) to ensure that updates being sent from the leader don't get missed 
> in between buffering and replication.
> This has the unfortunate side-effect of adding a 7-second pause to collection 
> creation whenever the replication factor is 2 or more, which slows down tests 
> massively - for example, DeleteReplicaTest takes about 54 seconds to execute 
> on my machine, 28 seconds of which is just pauses - over 50% of execution 
> time.  It's not actually possible to add documents to a collection before the 
> creation request has returned, so the recovery stage here isn't strictly 
> speaking necessary.  
> I think we could try adding a parameter to a CoreAdmin create request that 
> says the core is being created as part of a new collection, so it doesn't 
> need to try and recover from it's leader when it starts up.  Does this sound 
> sensible, or am I missing something here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

plan for getGlobalOrds->LongValues

[Yonik Seeley]

OrdinalMap.getGlobalOrds currently returns LongValues, which
was deprecated by the cutover to DV iterators (LongValues now extends
LegacyNumericDocValues).

An iterator API makes no sense for global ords, which are inherently
random-access since they are not based on docids.

It seems like we should either:
1) create a new LongValues abstract class with essentially the old
API:  long get(int index)
2) create a LongValues interface that can be optionally implemented by
classes wishing to expose a random access API

-Yonik

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Class#getResource returns null in JDK9 b140 if security manager is enabled (was: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!)

[Sean Mullan]

Sorry forgot to hit reply-all...

On 10/17/2016 01:33 PM, Sean Mullan wrote:

Weijun Wang is the best person to respond as he is the RE of JDK-8164705
- right now it is the middle of the night for him, but I would expect a
response from him once he comes online.

--Sean

On 10/16/2016 06:09 PM, Uwe Schindler wrote:

Hi again,

with jdk.io.permissionsUseCanonicalPath=true it also works, so it is
related to the new FilePermission code, so my first guess was true,
the issue is JDK-8164705.

Uwe


(I cc'ed jdk-dev@openjdk, reader there please read the previous mails
below, too).

I analyzed the problem, although I don't know exactly why it happens:
- On Windows it does not happen on my machine (no idea why!)
- On Linux it happens when tests are running with security manager
(this is
the default for Lucene and Jenkins does this)
- On Linux it does not happen if I run Lucene tests with "-
Dtests.useSecurityManager=false"

This makes me think it is related to this: "Remove pathname
canonicalization
from FilePermission" (https://bugs.openjdk.java.net/browse/JDK-8164705)

What seems to happen: The code calls Class.getResource to get back an
URL.
As the JAR file is somehow outside of the FilePermissions given to
the test
suite, it seems to fail. Maybe because some of the checks failed,
Class.getResource then returns a null reference, because it was not
able to
access the JAR file.

Were there some changes related to this: URLClassLoader and
FilePermission
checks?

How should we proceed?

Uwe

-
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/


-Original Message-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Sunday, October 16, 2016 10:10 PM
To: dev@lucene.apache.org
Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com; 'Muneer
Kolarkunnu' ; 'Dawid Weiss'

Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux
(32bit/jdk-9-ea+140) -
Build # 18064 - Unstable!

Hi,

I reverted the Lucene builds to build Java 9 138 for now. I will
later check if
this also happens with build 139, which I have to download first. I
will also
debug locally.

The code fails because this code hits "null" on getResource() at
morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:34)

https://github.com/morfologik/morfologik-
stemming/blob/master/morfologik-


polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32


This is impossible to happen, because the dict file is in same
package. I

have

no idea why this only fails here and not at other places in Lucene.
The main
difference looks like the use of URL instead of
getResourceAsStream() like
other places in Lucene.

So this seems to be a major regression in Java 9 build 140.

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


-Original Message-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Sunday, October 16, 2016 8:38 PM
To: dev@lucene.apache.org
Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com; 'Muneer
Kolarkunnu' ; 'Dawid Weiss'
; dev@lucene.apache.org
Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux
(32bit/jdk-9-ea+140) -
Build # 18064 - Unstable!

Hi,

this seems to be a new regression in Java 9 ea build 140.
Interestingly this
only affects 2 libraries (morphologic and commons-codec phonetic). We

use

loading of resources from classloaders at many places; it is
unclear to me,
why it only fails here. I will look into the code, but this is
outside of

Lucene.

I

think it might be some crazyness like using context class loader in
non-

proper

ways or similar.

Maybe it is a new bug in JDK 9 build 139 or build 140 (the last
working one
was build 138).

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


-Original Message-
From: Policeman Jenkins Server [mailto:jenk...@thetaphi.de]
Sent: Sunday, October 16, 2016 8:20 PM
To: dev@lucene.apache.org
Subject: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -

Build

#

18064 - Unstable!
Importance: Low

Build:
https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18064/
Java: 32bit/jdk-9-ea+140 -server -XX:+UseParallelGC

24 tests failed.
FAILED:






org.apache.lucene.analysis.morfologik.TestMorfologikAnalyzer.testSingleTok


ens

Error Message:
Could not read dictionary data.

Stack Trace:
java.lang.RuntimeException: Could not read dictionary data.
at






__randomizedtesting.SeedInfo.seed([27A360EA9CD9A4E8:A1C1C9729AE78F9

A]:0)
at


morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:38)

at






org.apache.lucene.analysis.morfologik.MorfologikAnalyzer.(Morfologik


Analyzer.java:52)
at






org.apache.lucene.analysis.morfologik.TestMorfologikAnalyzer.getTestAnaly


zer(TestMorfologikAnalyzer.java:39)
at

Re: Class#getResource returns null in JDK9 b140 if security manager is enabled (was: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!)

[Sean Mullan]

Weijun Wang is the best person to respond as he is the RE of JDK-8164705 
- right now it is the middle of the night for him, but I would expect a 
response from him once he comes online.


--Sean

On 10/16/2016 06:09 PM, Uwe Schindler wrote:

Hi again,

with jdk.io.permissionsUseCanonicalPath=true it also works, so it is related to 
the new FilePermission code, so my first guess was true, the issue is 
JDK-8164705.

Uwe


(I cc'ed jdk-dev@openjdk, reader there please read the previous mails
below, too).

I analyzed the problem, although I don't know exactly why it happens:
- On Windows it does not happen on my machine (no idea why!)
- On Linux it happens when tests are running with security manager (this is
the default for Lucene and Jenkins does this)
- On Linux it does not happen if I run Lucene tests with "-
Dtests.useSecurityManager=false"

This makes me think it is related to this: "Remove pathname canonicalization
from FilePermission" (https://bugs.openjdk.java.net/browse/JDK-8164705)

What seems to happen: The code calls Class.getResource to get back an URL.
As the JAR file is somehow outside of the FilePermissions given to the test
suite, it seems to fail. Maybe because some of the checks failed,
Class.getResource then returns a null reference, because it was not able to
access the JAR file.

Were there some changes related to this: URLClassLoader and FilePermission
checks?

How should we proceed?

Uwe

-
Uwe Schindler
uschind...@apache.org
ASF Member, Apache Lucene PMC / Committer
Bremen, Germany
http://lucene.apache.org/


-Original Message-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Sunday, October 16, 2016 10:10 PM
To: dev@lucene.apache.org
Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com; 'Muneer
Kolarkunnu' ; 'Dawid Weiss'

Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -
Build # 18064 - Unstable!

Hi,

I reverted the Lucene builds to build Java 9 138 for now. I will later check if
this also happens with build 139, which I have to download first. I will also
debug locally.

The code fails because this code hits "null" on getResource() at
morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:34)

https://github.com/morfologik/morfologik-
stemming/blob/master/morfologik-


polish/src/main/java/morfologik/stemming/polish/PolishStemmer.java#L32


This is impossible to happen, because the dict file is in same package. I

have

no idea why this only fails here and not at other places in Lucene. The main
difference looks like the use of URL instead of getResourceAsStream() like
other places in Lucene.

So this seems to be a major regression in Java 9 build 140.

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


-Original Message-
From: Uwe Schindler [mailto:u...@thetaphi.de]
Sent: Sunday, October 16, 2016 8:38 PM
To: dev@lucene.apache.org
Cc: dalibor.to...@oracle.com; balchandra.vai...@oracle.com; 'Muneer
Kolarkunnu' ; 'Dawid Weiss'
; dev@lucene.apache.org
Subject: RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -
Build # 18064 - Unstable!

Hi,

this seems to be a new regression in Java 9 ea build 140. Interestingly this
only affects 2 libraries (morphologic and commons-codec phonetic). We

use

loading of resources from classloaders at many places; it is unclear to me,
why it only fails here. I will look into the code, but this is outside of

Lucene.

I

think it might be some crazyness like using context class loader in non-

proper

ways or similar.

Maybe it is a new bug in JDK 9 build 139 or build 140 (the last working one
was build 138).

Uwe

-
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: u...@thetaphi.de


-Original Message-
From: Policeman Jenkins Server [mailto:jenk...@thetaphi.de]
Sent: Sunday, October 16, 2016 8:20 PM
To: dev@lucene.apache.org
Subject: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -

Build

#

18064 - Unstable!
Importance: Low

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18064/
Java: 32bit/jdk-9-ea+140 -server -XX:+UseParallelGC

24 tests failed.
FAILED:






org.apache.lucene.analysis.morfologik.TestMorfologikAnalyzer.testSingleTok

ens

Error Message:
Could not read dictionary data.

Stack Trace:
java.lang.RuntimeException: Could not read dictionary data.
at






__randomizedtesting.SeedInfo.seed([27A360EA9CD9A4E8:A1C1C9729AE78F9

A]:0)
at


morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:38)

at






org.apache.lucene.analysis.morfologik.MorfologikAnalyzer.(Morfologik

Analyzer.java:52)
at






org.apache.lucene.analysis.morfologik.TestMorfologikAnalyzer.getTestAnaly

zer(TestMorfologikAnalyzer.java:39)
at

Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!

[Alan Bateman]

On 17/10/2016 13:16, Uwe Schindler wrote:


Hi,

yes I checked more already: The issue is caused by the mentioned change 
(canonicalize of FilePermission). According to the docs of SecurityManager and 
FilePermission, code should always be able to read stuff below the classpath 
where the code was loaded from (in our case its part of a JAR file). So there 
is no need to add permissions for this, it should work automatically.

So the following code must work without any extra permissions:

URL url = this.getClass().getResource("somefilenexttoclassfile");
InputStream is = url.openStream();

Interestingly the first line already returns "null", means "resource not 
found", you don't get any SecurityException! As said before the code works without any 
problems if I pass the special JDK property jdk.io.permissionsUseCanonicalPath=true to the code. 
This is why I said that JDK-8164705 is causing the issue.

I will write a short reproducer and post it here. The code should work with 
SecurityManager enabled and empty policy file, as the resource is covered by 
the rule (everything below code source).

The getResourceXXX methods are specified to return null when denied by 
the security manager so you can't distinguish it from not found. If you 
can get trace output with -Djava.security.debug=failure,access then it 
might help diagnose this.


It's probably best to follow-up on security-dev rather than jdk9-dev as 
that is the mailing list where permission classes are maintained.


-Alan

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6187 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6187/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([2E82153B065AB7B4:463D2011D6C0A558]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:294)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-9481) BasicAuthPlugin should support standalone mode

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582693#comment-15582693
 ] 

Noble Paul commented on SOLR-9481:
--

+1 LGTM

> BasicAuthPlugin should support standalone mode
> --
>
> Key: SOLR-9481
> URL: https://issues.apache.org/jira/browse/SOLR-9481
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: authentication
> Attachments: SOLR-9481.patch, SOLR-9481.patch
>
>
> The BasicAuthPlugin currently only supports SolrCloud, and reads users and 
> credentials from ZK /security.json
> Add support for standalone mode operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9566) Can we avoid doing recovery when collections are first created?

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582651#comment-15582651
 ] 

Alan Woodward commented on SOLR-9566:
-

I got a verbal +1 from [~markrmil...@gmail.com] at Revolution, so I'll commit 
this shortly.  Anybody who disagrees should speak up now...

> Can we avoid doing recovery when collections are first created?
> ---
>
> Key: SOLR-9566
> URL: https://issues.apache.org/jira/browse/SOLR-9566
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
> Attachments: SOLR-9566.patch
>
>
> When a core starts up as part of a collection, and it's not a shard leader, 
> it goes into recovery, part of which involves a 7 second wait (set in 
> SOLR-7141) to ensure that updates being sent from the leader don't get missed 
> in between buffering and replication.
> This has the unfortunate side-effect of adding a 7-second pause to collection 
> creation whenever the replication factor is 2 or more, which slows down tests 
> massively - for example, DeleteReplicaTest takes about 54 seconds to execute 
> on my machine, 28 seconds of which is just pauses - over 50% of execution 
> time.  It's not actually possible to add documents to a collection before the 
> creation request has returned, so the recovery stage here isn't strictly 
> speaking necessary.  
> I think we could try adding a parameter to a CoreAdmin create request that 
> says the core is being created as part of a new collection, so it doesn't 
> need to try and recover from it's leader when it starts up.  Does this sound 
> sensible, or am I missing something here?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-575) Highlighting spans should merge across phrase query

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582649#comment-15582649
 ] 

David Smiley commented on SOLR-575:
---

It's a "wish-list" feature.  It could happen; leave it open.  IMO I don't think 
wish-list features should be assigned to a particular fix-version it just 
leads to "Move issue to" etc. comments on each release.

It was nigh impossible against the standard {{Highlighter.java}} for which this 
issue was created against back in 2008.  At the time we had only one 
highlighter. Now we have highlighters where this feature request is plausible.

> Highlighting spans should merge across phrase query
> ---
>
> Key: SOLR-575
> URL: https://issues.apache.org/jira/browse/SOLR-575
> Project: Solr
>  Issue Type: Improvement
>  Components: highlighter
>Reporter: Brian Whitman
>Priority: Minor
>
> Somewhat related to but separate from SOLR-553,
> It would be nice if the highlighter component "joined" the formatter tags 
> across an entire PhraseQuery.
> e.g. 
> Lights (Live) : I Love You But 
> I've Chosen Darkness :
> should really be
> Lights (Live) : I Love You But I've Chosen Darkness :
> assuming the query that generated these fragments was "I Love You But I've 
> Chosen Darkness"
> I assume there's issues with stopwords (the But in the name was not formatted)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9255) Start Script Basic Authentication

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-9255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated SOLR-9255:
--
Attachment: SOLR-9255.patch

[~MartinLoeper], I changed "Affects Version" to master, do you agree? I'm also 
attaching a first patch. It also adds auth support to solr.cmd.

Not yet tested

> Start Script Basic Authentication
> -
>
> Key: SOLR-9255
> URL: https://issues.apache.org/jira/browse/SOLR-9255
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Authentication
>Affects Versions: master (7.0)
>Reporter: Martin Löper
>Assignee: Jan Høydahl
> Fix For: master (7.0)
>
> Attachments: SOLR-9255.patch
>
>
> I configured SSL and BasicAuthentication with Rule-Based-Authorization.
> I noticed that since the latest changes from 6.0.1 to 6.1.0 I cannot pass the 
> Basic Authentication Credentials to the Solr Start Script anymore. For the 
> previous release I did this via the bin/solr.in.sh shellscript.
> What has happened with the SOLR_AUTHENTICATION_CLIENT_CONFIGURER and 
> SOLR_AUTHENTICATION_OPTS parameters? Are they still in use or is there a new 
> way to pass basic auth credentials on the command-line?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7500) Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582614#comment-15582614
 ] 

David Smiley commented on LUCENE-7500:
--

Yeah; that would be the simplest change to get the biggest benefit on the part 
of the API closer to the surface (granted all this is fairly low-level).

> Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)
> --
>
> Key: LUCENE-7500
> URL: https://issues.apache.org/jira/browse/LUCENE-7500
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: David Smiley
> Fix For: master (7.0)
>
>
> {{Fields}} seems like a pointless intermediary between the {{LeafReader}} and 
> {{Terms}}. Why not have {{LeafReader.getTerms(fieldName)}} instead? One loses 
> the ability to get the count and iterate over indexed fields, but it's not 
> clear what real use-cases are for that and such rare needs could figure that 
> out with FieldInfos.
> [~mikemccand] pointed out that we'd probably need to re-introduce a 
> {{TermVectors}} class since TV's are row-oriented not column-oriented.  IMO 
> they should be column-oriented but that'd be a separate issue.
> _(p.s. I'm lacking time to do this w/i the next couple months so if someone 
> else wants to tackle it then great)_



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582608#comment-15582608
 ] 

Yonik Seeley commented on SOLR-9642:


Looks good to me. I'll commit after some testing...

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

RE: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!

[Uwe Schindler]

Hi,

> One more thing:
> 
> You said "(although it should not be needed). Can you confirm that? I mean,
> if you remove that line, does the test still passes with -
> Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into
> other reasons, since I did see in many other places file permission on
> /home/jenkins//morfologik-polish-2.1.0.jar itself is granted.

See my other mail! It should not be needed, because Class#getResource does not 
throw or document any Security implications. But in fact if you don't wrap a 
doPrivileged around the call, it will also fail without our extra permission. 
It was added to work around this issue with 3rd party JARs.

I just blew up with the symlink on home dir.

Uwe

> Thanks
> Max
> 
> 
> > On Oct 17, 2016, at 11:17 PM, Wang Weijun 
> wrote:
> >
> > Yes, this is where the problem is.
> >
> > So it looks like the permission is granted in a policy file instead of being
> granted by the class loader itself. In this case, the path of the permission
> must match how you access that file.
> >
> > I'll think about your suggestion. However, can you guarantee the code
> always accesses the file using the canonicalized path? For example, suppose
> the actual file is /x, both /a and /b symlink to it, and you grant the 
> permission
> on /a in a policy file. Even if I canonicalize /a to /x and grant permissions 
> on
> both /a and /x, you will still see a failure if the code access /b.
> >
> > --Max
> >
> >> On Oct 17, 2016, at 11:03 PM, Uwe Schindler 
> wrote:
> >>
> >> Hi,
> >>
> >> I attached the debugging logs ("all") for the working (conaonicalized) and
> non-working case to this mail. Please keep in mind that around the
> getResource() is no doPrivileged, so it looks like it uses privileges of 
> outer JAR.
> >>
> >> I also checked, we have the following explicitely in our policy file, which
> explicitly allows everything in the IVY cache (although it should not be
> needed) - so it should really work anyways!!!
> >>
> >> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-",
> "read";
> >>
> >> But I think I know the issue - which is really stupid - but it’s a fact 
> >> affecting
> mny people! Here is what the policy file parsing says above the
> permission given before (please compare first line and second line):
> >>
> >>  [junit4]   2> Active Principals: []
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/var/lib/jenkins/.ivy2/cache/-" "read")
> >>  [junit4]   2> policy:   granting ("java.io.FilePermission"
> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
> >>
> >> Unfortunately, the user changed its home directory, the old one
> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not
> updated. So ${user.home} looks different. This is the reason why it did not
> happen locally on my windows machine!
> >>
> >> IMHO: When reading the policy file, it should canonicalize all paths,
> especially stuff like ${user.home}. At runtime when actually checking the
> permissions it should not canonicalize for performance reasons. Not sure if
> this is a bug, but this could hit lots of people. I know that other Jenkins
> servers where Lucene runs their tests (not yet on Java 9) are setup in similar
> ways!
> >>
> >> Uwe
> >>
> >> -
> >> Uwe Schindler
> >> H.-H.-Meier-Allee 63, D-28213 Bremen
> >> http://www.thetaphi.de
> >> eMail: u...@thetaphi.de
> >>
> >>> -Original Message-
> >>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
> >>> Sent: Monday, October 17, 2016 4:03 PM
> >>> To: dev@lucene.apache.org
> >>> Cc: Dalibor Topic ; Balchandra Vaidya
> >>> ; Muneer Kolarkunnu
> >>> ; Rory O'Donnell
> >>> 
> >>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140)
> -
> >>> Build # 18064 - Unstable!
> >>>
> >>> OK, I'll try it tomorrow.
> >>>
> >>> At the same time can you try -Djava.security.debug=all or -
> >>> Djava.security.debug=scl and see if it shows what permission is granted?
> >>>
> >>> Thanks
> >>> Max
> >>>
>  On Oct 17, 2016, at 9:07 PM, Uwe Schindler 
> >>> wrote:
> 
>  FYI,
> 
>  With:
>  $ ant test -Dtests.useSecurityManager=true -Dargs='-
> >>> Djava.security.debug=access -
> Djdk.io.permissionsUseCanonicalPath=true'
> 
>  I can see the following line:
> 
>  [junit4]   2> access: access allowed ("java.io.FilePermission"
> >>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
> >>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
> 
>  So there is a difference!!!
> 
>  Uwe
> 
>  -
>  Uwe Schindler
>  uschind...@apache.org
>  ASF Member, Apache Lucene 

[jira] [Commented] (SOLR-9481) BasicAuthPlugin should support standalone mode

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582583#comment-15582583
 ] 

Noble Paul commented on SOLR-9481:
--

Sure , will do

> BasicAuthPlugin should support standalone mode
> --
>
> Key: SOLR-9481
> URL: https://issues.apache.org/jira/browse/SOLR-9481
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: security
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>  Labels: authentication
> Attachments: SOLR-9481.patch, SOLR-9481.patch
>
>
> The BasicAuthPlugin currently only supports SolrCloud, and reads users and 
> credentials from ZK /security.json
> Add support for standalone mode operation



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-9642) Refactor the core level snapshot cleanup mechanism to rely on Lucene

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley reassigned SOLR-9642:
--

Assignee: Yonik Seeley

> Refactor the core level snapshot cleanup mechanism to rely on Lucene
> 
>
> Key: SOLR-9642
> URL: https://issues.apache.org/jira/browse/SOLR-9642
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.2
>Reporter: Hrishikesh Gadre
>Assignee: Yonik Seeley
>
> SOLR-9269 introduced a mechanism to create/delete snapshots for a Solr core 
> (using Lucene IndexDeletionPolicy). The current snapshot cleanup mechanism is 
> based on reference counting the index files shared between multiple segments. 
> Since this mechanism completely skips the Lucene APIs, it is not portable 
> (e.g. it doesn't work on 4.10.3 version).
> I propose an alternative implementation which relies exclusively on Lucene 
> IndexWriter (+ IndexDeletionPolicy) for cleanup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) - Build # 18064 - Unstable!

[Wang Weijun]

One more thing:

You said "(although it should not be needed). Can you confirm that? I mean, if 
you remove that line, does the test still passes with 
-Djdk.io.permissionsUseCanonicalPath=true? If so, we will need to look into 
other reasons, since I did see in many other places file permission on 
/home/jenkins//morfologik-polish-2.1.0.jar itself is granted.

Thanks
Max


> On Oct 17, 2016, at 11:17 PM, Wang Weijun  wrote:
> 
> Yes, this is where the problem is.
> 
> So it looks like the permission is granted in a policy file instead of being 
> granted by the class loader itself. In this case, the path of the permission 
> must match how you access that file.
> 
> I'll think about your suggestion. However, can you guarantee the code always 
> accesses the file using the canonicalized path? For example, suppose the 
> actual file is /x, both /a and /b symlink to it, and you grant the permission 
> on /a in a policy file. Even if I canonicalize /a to /x and grant permissions 
> on both /a and /x, you will still see a failure if the code access /b.
> 
> --Max
> 
>> On Oct 17, 2016, at 11:03 PM, Uwe Schindler  wrote:
>> 
>> Hi,
>> 
>> I attached the debugging logs ("all") for the working (conaonicalized) and 
>> non-working case to this mail. Please keep in mind that around the 
>> getResource() is no doPrivileged, so it looks like it uses privileges of 
>> outer JAR. 
>> 
>> I also checked, we have the following explicitely in our policy file, which 
>> explicitly allows everything in the IVY cache (although it should not be 
>> needed) - so it should really work anyways!!!
>> 
>> permission java.io.FilePermission "${user.home}${/}.ivy2${/}cache${/}-", 
>> "read";
>> 
>> But I think I know the issue - which is really stupid - but it’s a fact 
>> affecting mny people! Here is what the policy file parsing says above 
>> the permission given before (please compare first line and second line):
>> 
>>  [junit4]   2>   Active Principals: []
>>  [junit4]   2> policy:   granting ("java.io.FilePermission" 
>> "/home/jenkins/workspace/Lucene-Solr-master-Linux/lucene/-" "read")
>>  [junit4]   2> policy:   granting ("java.io.FilePermission" 
>> "/var/lib/jenkins/.ivy2/cache/-" "read")
>>  [junit4]   2> policy:   granting ("java.io.FilePermission" 
>> "/home/jenkins/tools/java/64bit/jdk-9-ea+140/-" "read,execute")
>> 
>> Unfortunately, the user changed its home directory, the old one 
>> (var/lib/jenkins) symlinked to the new one, but /etc/passwd was not updated. 
>> So ${user.home} looks different. This is the reason why it did not happen 
>> locally on my windows machine!
>> 
>> IMHO: When reading the policy file, it should canonicalize all paths, 
>> especially stuff like ${user.home}. At runtime when actually checking the 
>> permissions it should not canonicalize for performance reasons. Not sure if 
>> this is a bug, but this could hit lots of people. I know that other Jenkins 
>> servers where Lucene runs their tests (not yet on Java 9) are setup in 
>> similar ways!
>> 
>> Uwe
>> 
>> -
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: u...@thetaphi.de
>> 
>>> -Original Message-
>>> From: Wang Weijun [mailto:weijun.w...@oracle.com]
>>> Sent: Monday, October 17, 2016 4:03 PM
>>> To: dev@lucene.apache.org
>>> Cc: Dalibor Topic ; Balchandra Vaidya
>>> ; Muneer Kolarkunnu
>>> ; Rory O'Donnell
>>> 
>>> Subject: Re: [JENKINS-EA] Lucene-Solr-master-Linux (32bit/jdk-9-ea+140) -
>>> Build # 18064 - Unstable!
>>> 
>>> OK, I'll try it tomorrow.
>>> 
>>> At the same time can you try -Djava.security.debug=all or -
>>> Djava.security.debug=scl and see if it shows what permission is granted?
>>> 
>>> Thanks
>>> Max
>>> 
 On Oct 17, 2016, at 9:07 PM, Uwe Schindler 
>>> wrote:
 
 FYI,
 
 With:
 $ ant test -Dtests.useSecurityManager=true -Dargs='-
>>> Djava.security.debug=access -Djdk.io.permissionsUseCanonicalPath=true'
 
 I can see the following line:
 
 [junit4]   2> access: access allowed ("java.io.FilePermission"
>>> "/home/jenkins/.ivy2/cache/org.carrot2/morfologik-
>>> polish/bundles/morfologik-polish-2.1.0.jar" "read")
 
 So there is a difference!!!
 
 Uwe
 
 -
 Uwe Schindler
 uschind...@apache.org
 ASF Member, Apache Lucene PMC / Committer
 Bremen, Germany
 http://lucene.apache.org/
 
> -Original Message-
> From: Uwe Schindler [mailto:uschind...@apache.org]
> Sent: Monday, October 17, 2016 2:44 PM
> To: dev@lucene.apache.org; 'Wang Weijun' 
> Cc: 'Dalibor Topic' ; 'Balchandra Vaidya'
> ; 'Muneer Kolarkunnu'
> ; 'Rory O'Donnell'
> 

[jira] [Commented] (SOLR-575) Highlighting spans should merge across phrase query

[Alexandre Rafalovitch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582430#comment-15582430
 ] 

Alexandre Rafalovitch commented on SOLR-575:


Ok, good to know it is a live one.

So, what's the next action? Leave it as a suggestion against master? Or 
something more specific?

> Highlighting spans should merge across phrase query
> ---
>
> Key: SOLR-575
> URL: https://issues.apache.org/jira/browse/SOLR-575
> Project: Solr
>  Issue Type: Improvement
>  Components: highlighter
>Reporter: Brian Whitman
>Priority: Minor
>
> Somewhat related to but separate from SOLR-553,
> It would be nice if the highlighter component "joined" the formatter tags 
> across an entire PhraseQuery.
> e.g. 
> Lights (Live) : I Love You But 
> I've Chosen Darkness :
> should really be
> Lights (Live) : I Love You But I've Chosen Darkness :
> assuming the query that generated these fragments was "I Love You But I've 
> Chosen Darkness"
> I assume there's issues with stopwords (the But in the name was not formatted)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+138) - Build # 18071 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18071/
Java: 64bit/jdk-9-ea+138 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.lucene.analysis.morfologik.TestMorfologikFilterFactory.testDefaultDictionary

Error Message:
Could not read dictionary data.

Stack Trace:
java.lang.RuntimeException: Could not read dictionary data.
at 
__randomizedtesting.SeedInfo.seed([9C60655C3B12AD27:CC2462D1F50C57BE]:0)
at 
morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:38)
at 
org.apache.lucene.analysis.morfologik.MorfologikFilterFactory.inform(MorfologikFilterFactory.java:85)
at 
org.apache.lucene.analysis.morfologik.TestMorfologikFilterFactory.testDefaultDictionary(TestMorfologikFilterFactory.java:56)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:535)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
Caused by: java.io.IOException: Polish dictionary resource not found.
at 
morfologik.stemming.polish.PolishStemmer.(PolishStemmer.java:34)
... 38 more




Build Log:
[...truncated 62546 lines...]


-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: 

[jira] [Commented] (LUCENE-7500) Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7500?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582426#comment-15582426
 ] 

Adrien Grand commented on LUCENE-7500:
--

I just looked again at how Fields are used, and I'd be tempted to just remove 
{{LeafReader.fields()}}. Consumers would have to call 
{{LeafReader.terms(String)}} instead. Fields would still be useful in 
{{CodecReader.getPostingsReader()}} (consistently with {{getNormsReader}}, 
{{getPointsReader}}, {{getDocValuesReader}} which all deal with all fields at 
once) and {{LeafReader.getTermVectors(int)}} for term vectors.



> Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)
> --
>
> Key: LUCENE-7500
> URL: https://issues.apache.org/jira/browse/LUCENE-7500
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: David Smiley
> Fix For: master (7.0)
>
>
> {{Fields}} seems like a pointless intermediary between the {{LeafReader}} and 
> {{Terms}}. Why not have {{LeafReader.getTerms(fieldName)}} instead? One loses 
> the ability to get the count and iterate over indexed fields, but it's not 
> clear what real use-cases are for that and such rare needs could figure that 
> out with FieldInfos.
> [~mikemccand] pointed out that we'd probably need to re-introduce a 
> {{TermVectors}} class since TV's are row-oriented not column-oriented.  IMO 
> they should be column-oriented but that'd be a separate issue.
> _(p.s. I'm lacking time to do this w/i the next couple months so if someone 
> else wants to tackle it then great)_



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582392#comment-15582392
 ] 

Jan Høydahl commented on SOLR-8370:
---

Alexandre, would you care to test the JS bug with a plain trunk version and see 
if you can reproduce without this patch, and file a JIRA if necessary?

I'm changing wording from Default -> Global
If the other issues are unrelated to this one, then are there more stuff to 
change for this JIRA?

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, screenshot-1.png, 
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582387#comment-15582387
 ] 

Jan Høydahl commented on SOLR-8370:
---

bq. In the global similarity section, The tooltip seems to always say: 
o.a.s.search.similarities.SchemaSimilarityFactory$1 whatever similarity I 
configure. The field similarity tooltip seems correct.

I tested with {{}} and it 
displays correctly the fieldName


> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, screenshot-1.png, 
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-575) Highlighting spans should merge across phrase query

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated SOLR-575:
--
Affects Version/s: (was: 1.2)
Fix Version/s: (was: 6.0)
   (was: 4.9)

It's "UnifiedHighlighter" but I like your name too :-)

The issue here is valid -- AFAIK no highlighter works as the o.p. wants it to.  
I can imagine how the UH could be modified to work this way... -- by having 
PhraseHelper produce one PostingsEnum per SpanQuery (what a phrase is converted 
into) instead of wrapping/filtering PE's given to it.  Then, instead of 
producing one position per underlying-term, it could produce one virtual 
position of the span.  This would only be applicable to phrase queries without 
slop; not other position sensitive queries or span queries generally.  It might 
improve passage relevance too.

> Highlighting spans should merge across phrase query
> ---
>
> Key: SOLR-575
> URL: https://issues.apache.org/jira/browse/SOLR-575
> Project: Solr
>  Issue Type: Improvement
>  Components: highlighter
>Reporter: Brian Whitman
>Priority: Minor
>
> Somewhat related to but separate from SOLR-553,
> It would be nice if the highlighter component "joined" the formatter tags 
> across an entire PhraseQuery.
> e.g. 
> Lights (Live) : I Love You But 
> I've Chosen Darkness :
> should really be
> Lights (Live) : I Love You But I've Chosen Darkness :
> assuming the query that generated these fragments was "I Love You But I've 
> Chosen Darkness"
> I assume there's issues with stopwords (the But in the name was not formatted)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7494) Explore making PointValues a per-field API like doc values

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582376#comment-15582376
 ] 

Adrien Grand commented on LUCENE-7494:
--

The diff stats report a reduction of the number of lines indeed:

{code}
 51 files changed, 648 insertions(+), 958 deletions(-)
{code}

> Explore making PointValues a per-field API like doc values
> --
>
> Key: LUCENE-7494
> URL: https://issues.apache.org/jira/browse/LUCENE-7494
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7494.patch
>
>
> This is a follow-up to LUCENE-7491. Maybe we could simplify things a bit by 
> changing {{LeafReader.getPointValues()}} to 
> {{LeafReader.getPointValues(String fieldName)}} and removing all {{String 
> fieldName}} parameters from {{PointValues}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8370) Display Similarity Factory in use in Schema-Browser

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-8370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582370#comment-15582370
 ] 

Jan Høydahl commented on SOLR-8370:
---

Looking at author_s, it is broken from Luke. The endpoint 
http://localhost:8983/solr/techproducts/admin/luke does not contain this 
dynamic field (but it contains address_s), neither does while 
http://localhost:8983/solr/techproducts/admin/luke?show=schema but it is listed 
as a copyDest. 

I'm not sure if Luke was supposed to bring it back as a field, but tell us that 
it is a dynField, or whether {{mergeIndexAndSchemaData(index, schema.schema);}} 
method in schema.js was supposed to find out that by itself, as it is now, it 
is that method that populates {{author_s}} but in a partial:true state... It 
also has a line 
{code}
display.partialState = is.field && !!data.fields[name].partial;
{code}
which seems to set a variabel but do nothing further with it. So there may be 
some work in progress here that was never finalised? This also smells like a 
new JIRA, not part of this one...

> Display Similarity Factory in use in Schema-Browser
> ---
>
> Key: SOLR-8370
> URL: https://issues.apache.org/jira/browse/SOLR-8370
> Project: Solr
>  Issue Type: Improvement
>  Components: UI
>Reporter: Jan Høydahl
>Assignee: Jan Høydahl
>Priority: Trivial
>  Labels: newdev
> Fix For: 6.3, master (7.0)
>
> Attachments: SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, 
> SOLR-8370.patch, SOLR-8370.patch, SOLR-8370.patch, screenshot-1.png, 
> screenshot-2.png, screenshot-3.png, screenshot-4.png
>
>
> Perhaps the Admin UI Schema browser should also display which 
> {{}} that is in use in schema, like it does per-field.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7494) Explore making PointValues a per-field API like doc values

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582351#comment-15582351
 ] 

David Smiley commented on LUCENE-7494:
--

+1 to the patch. I like how it, at least it seems, like there is a net 
reduction in code.

> Explore making PointValues a per-field API like doc values
> --
>
> Key: LUCENE-7494
> URL: https://issues.apache.org/jira/browse/LUCENE-7494
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7494.patch
>
>
> This is a follow-up to LUCENE-7491. Maybe we could simplify things a bit by 
> changing {{LeafReader.getPointValues()}} to 
> {{LeafReader.getPointValues(String fieldName)}} and removing all {{String 
> fieldName}} parameters from {{PointValues}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7500) Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)

[David Smiley (JIRA)]

David Smiley created LUCENE-7500:


 Summary: Nuke Fields.java in lieu of LeafReader.getTerms(fieldName)
 Key: LUCENE-7500
 URL: https://issues.apache.org/jira/browse/LUCENE-7500
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: David Smiley
 Fix For: master (7.0)


{{Fields}} seems like a pointless intermediary between the {{LeafReader}} and 
{{Terms}}. Why not have {{LeafReader.getTerms(fieldName)}} instead? One loses 
the ability to get the count and iterate over indexed fields, but it's not 
clear what real use-cases are for that and such rare needs could figure that 
out with FieldInfos.

[~mikemccand] pointed out that we'd probably need to re-introduce a 
{{TermVectors}} class since TV's are row-oriented not column-oriented.  IMO 
they should be column-oriented but that'd be a separate issue.

_(p.s. I'm lacking time to do this w/i the next couple months so if someone 
else wants to tackle it then great)_



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9077) Streaming expression in solr doesnot support collection alias

[Dennis Gove (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582274#comment-15582274
 ] 

Dennis Gove commented on SOLR-9077:
---

It might be necessary to do that in each source stream, though a centralized 
location might be better. The StreamFactory could be a place to store the 
aliases after looking them up but I'm not 100% sure that's the best place.

> Streaming expression in solr doesnot support collection alias
> -
>
> Key: SOLR-9077
> URL: https://issues.apache.org/jira/browse/SOLR-9077
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.5.1
>Reporter: Suds
>Priority: Minor
>
> Streaming expression in solr does not support collection alias
> when I tried to access collection alias I get null pointer exception 
> issue seems to be related to following code , clusterState.getActiveSlices 
> returns null 
>  Collection slices = clusterState.getActiveSlices(this.collection);
>  for(Slice slice : slices) {
> }
> fix seems to fairly simple , clusterState.getActiveSlices can be made aware 
> of collection alias. I am not sure what will happen when we have large alias 
> which has hundred of slices.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7494) Explore making PointValues a per-field API like doc values

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-7494:
-
Attachment: LUCENE-7494.patch

Here is a patch that deals with points. I'd like to do changes to Fields / term 
vectors in a different issue.

> Explore making PointValues a per-field API like doc values
> --
>
> Key: LUCENE-7494
> URL: https://issues.apache.org/jira/browse/LUCENE-7494
> Project: Lucene - Core
>  Issue Type: Wish
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-7494.patch
>
>
> This is a follow-up to LUCENE-7491. Maybe we could simplify things a bit by 
> changing {{LeafReader.getPointValues()}} to 
> {{LeafReader.getPointValues(String fieldName)}} and removing all {{String 
> fieldName}} parameters from {{PointValues}}?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-3953) Using many fields in filters, sorts causes OOM

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-3953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-3953.
--
   Resolution: Fixed
Fix Version/s: master (7.0)

This looks mostly due to FieldCache storing data in a dense way. Now that 
FieldCache has been replaced with doc values, you would need many more fields 
in order to trigger OOMEs, but disk usage and indexing speed would still suffer 
from these sparse fields. I think we can consider it as fixed in master (7.0) 
thanks to the new sparse APIs for doc values and the new doc values format.

That said, it is still advised to have dense fields whenever possible. We made 
sparse fields more cost-efficient, but there is still a penalty compared to 
dense fields.

> Using many fields in filters, sorts causes OOM
> --
>
> Key: LUCENE-3953
> URL: https://issues.apache.org/jira/browse/LUCENE-3953
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: core/search
>Affects Versions: 4.0-ALPHA
>Reporter: Des Lownds
> Fix For: master (7.0)
>
> Attachments: FieldCacheLimitPatch.patch
>
>
> I have an index with ~1M documents, and many uniquely named fields. Each 
> document only has a few fields populated, but there is a large set of dynamic 
> fields across all docs. In other words, theres many sparse fields.
> Eventually the jvm will fall over due to OOM errors, because lucene's 
> fieldCache grows without bounds. Adding more heap does not help.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-1560) make "optimize" do partial optimizes under the covers

[Alexandre Rafalovitch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-1560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexandre Rafalovitch closed SOLR-1560.
---
Resolution: Not A Problem

Nobody confirmed if this still relevant in more than a year. I suspect it is 
not with a lot of changes since this issue was opened.

> make "optimize" do partial optimizes under the covers
> -
>
> Key: SOLR-1560
> URL: https://issues.apache.org/jira/browse/SOLR-1560
> Project: Solr
>  Issue Type: Improvement
>  Components: update
>Reporter: Hoss Man
>
> In Lucene, an optimize() call iteratively merges segments until only one is 
> left - and while it's merging it (ultimately) needs to make a copy of the 
> entire index, because readers attempting to open the index "mid-optimize" 
> need to see a consistent copy of the index.
> In Solr, we have control over when new readers/searchers get opened, so what 
> if when we recieved an  command, we under the covers we made 
> iterative partial optimize calls and only opened a new searcher when we were 
> finished with all of them?  In theory this seems like  it would help reduce 
> the disk space used during optimize, without really affecting the time it 
> takes to "optimize"
> These are the threads that prompted this idea...
> http://old.nabble.com/eternal-optimize-interrupted-to24805680.html#a24805680
> http://old.nabble.com/Re%3A-eternal-optimize-interrupted-to24928754.html#a24928754
> http://old.nabble.com/Optimization-of-large-shard-succeeded-to25809281.html#a25809281



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org