[jira] [Commented] (SOLR-9718) All JUnits of Streaming use java assert to compare double should use assertEquals()

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663018#comment-15663018
 ] 

ASF subversion and git services commented on SOLR-9718:
---

Commit 3147c5c423563850cf4c96ed645b5f24d07cd932 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3147c5c ]

SOLR-9718: replace assert and assertTrue() to assertEquals()


> All JUnits of Streaming use java assert to compare double should use 
> assertEquals()
> ---
>
> Key: SOLR-9718
> URL: https://issues.apache.org/jira/browse/SOLR-9718
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> get rid of java {{assert}} altogether. it does not give any messages if there 
> are any failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7466) Allow optional leading wildcards in complexphrase

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663014#comment-15663014
 ] 

Mikhail Khludnev commented on SOLR-7466:


Or how to move ReversedWildcardFilterFactory into Lucene and let Lucene's query 
parser to detect it in analysis chain like it's done in  
o.a.solr.p.SolrQueryParserBase.getReversedWildcardFilterFactory(FieldType) ? 
Isn't there anything in Lucene which gets on par with Solr's one? 

> Allow optional leading wildcards in complexphrase
> -
>
> Key: SOLR-7466
> URL: https://issues.apache.org/jira/browse/SOLR-7466
> Project: Solr
>  Issue Type: Improvement
>  Components: query parsers
>Affects Versions: 4.8
>Reporter: Andy hardin
>  Labels: complexPhrase, query-parser, wildcards
>
> Currently ComplexPhraseQParser (SOLR-1604) allows trailing wildcards on terms 
> in a phrase, but does not allow leading wildcards.  I would like the option 
> to be able to search for terms with both trailing and leading wildcards.  
> For example with:
> {!complexphrase allowLeadingWildcard=true} "j* *th"
> would match "John Smith", "Jim Smith", but not "John Schmitt"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9718) All JUnits of Streaming use java assert to compare double should use assertEquals()

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663015#comment-15663015
 ] 

ASF subversion and git services commented on SOLR-9718:
---

Commit 907bed887d6be1aaec832c4f6d395d051bb49b17 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=907bed8 ]

SOLR-9718: replace assert and assertTrue() to assertEquals()


> All JUnits of Streaming use java assert to compare double should use 
> assertEquals()
> ---
>
> Key: SOLR-9718
> URL: https://issues.apache.org/jira/browse/SOLR-9718
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> get rid of java {{assert}} altogether. it does not give any messages if there 
> are any failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9718) All JUnits of Streaming use java assert to compare double should use assertEquals()

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15663002#comment-15663002
 ] 

ASF subversion and git services commented on SOLR-9718:
---

Commit a32b0495981997920c7988d4b1b93f5b85d3e553 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=a32b049 ]

SOLR-9718: replace assert and assertTrue() to assertEquals()


> All JUnits of Streaming use java assert to compare double should use 
> assertEquals()
> ---
>
> Key: SOLR-9718
> URL: https://issues.apache.org/jira/browse/SOLR-9718
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> get rid of java {{assert}} altogether. it does not give any messages if there 
> are any failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9718) All JUnits of Streaming use java assert to compare double should use assertEquals()

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662997#comment-15662997
 ] 

ASF subversion and git services commented on SOLR-9718:
---

Commit b359636fd9f272f3617a97a276fa41f7dd252016 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=b359636 ]

SOLR-9718: replace assert and assertTrue() to assertEquals()


> All JUnits of Streaming use java assert to compare double should use 
> assertEquals()
> ---
>
> Key: SOLR-9718
> URL: https://issues.apache.org/jira/browse/SOLR-9718
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> get rid of java {{assert}} altogether. it does not give any messages if there 
> are any failures



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662936#comment-15662936
 ] 

David Smiley commented on SOLR-9708:


_(see github comments)_
Looking good Tim; just some super minor stuff.

It's unfortunate that users need to go into their solrconfig to enable this 
highlighter.  What if there was a boolean param {{hl.useUnifiedHighlighter}} 
that was evaluated at the very beginning of 
{{DefaultSolrHighlighter.doHighlighting}} that delegated to it?  I know from a 
code perspective it's _very_ hacky but consider the user experience.  In Solr 
we're trying to push towards avoiding the need to touch configs.  Fortunately, 
this highlighter is completely configured from params (not so for SH & FVH)... 
except for enabling it in the first place (which we can change here). The DSH 
after all already handles FVH.  In a future issue for 7.0 I'd like to propose 
some re-arrangement to the Solr highlighting code to simplify & clean up things.

> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662927#comment-15662927
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87741032
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java ---
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import java.io.IOException;
+import java.text.BreakIterator;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.postingshighlight.WholeBreakIterator;
+import org.apache.lucene.search.uhighlight.DefaultPassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageScorer;
+import org.apache.lucene.search.uhighlight.UnifiedHighlighter;
+import org.apache.solr.common.params.HighlightParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.core.PluginInfo;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.request.SolrRequestInfo;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.RTimerTree;
+import org.apache.solr.util.plugin.PluginInfoInitialized;
+
+/*
+ * TODO: The HighlightComponent should not call rewrite on the query; it 
should be up to the
+ * SolrHighlighter to do if needed.  Furthermore this arrangement is odd 
-- why are these abstractions separate?
+ */
+
+/**
+ * Highlighter impl that uses {@link UnifiedHighlighter}
+ * 
+ * Example configuration with default values:
+ * 
+ * requestHandler name="standard" 
class="solr.StandardRequestHandler"
+ * lst name="defaults"
+ * int name="hl.snippets"1/int
+ * str name="hl.tag.pre"lt;emgt;/str
+ * str name="hl.tag.post"lt;/emgt;/str
+ * str name="hl.tag.ellipsis"... /str
+ * bool name="hl.defaultSummary"true/bool
+ * str name="hl.encoder"simple/str
+ * float name="hl.score.k1"1.2/float
+ * float name="hl.score.b"0.75/float
+ * float name="hl.score.pivot"87/float
+ * str name="hl.bs.language"/str
+ * str name="hl.bs.country"/str
+ * str name="hl.bs.variant"/str
+ * str name="hl.bs.type"SENTENCE/str
+ * int name="hl.maxAnalyzedChars"1/int
+ * bool name="hl.highlightMultiTerm"true/bool
+ * /lst
+ * /requestHandler
+ * 
+ * ...
+ * 
+ * searchComponent class="solr.HighlightComponent" name="highlight"
+ * highlighting 
class="org.apache.solr.highlight.UnifiedSolrHighlighter"/
+ * /searchComponent
+ * 
+ * 
+ * Notes:
+ * 
+ * hl.q (string) can specify the query
+ * hl.fl (string) specifies the field list.
+ * hl.snippets (int) specifies how many snippets to return.
+ * hl.tag.pre (string) specifies text which appears before a 
highlighted term.
+ * hl.tag.post (string) specifies text which appears after a 
highlighted term.
+ * hl.tag.ellipsis (string) specifies text which joins non-adjacent 
passages. The default is to retain each
+ * value in a list without joining them.
+ * hl.defaultSummary (bool) specifies if a field should have a default 
summary of the leading text.
+ * hl.encoder (string) can be 'html' (html escapes content) or 
'simple' (no escaping).
+ * hl.score.k1 (float) specifies bm25 

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662931#comment-15662931
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740491
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
+public class TestUnifiedSolrHighlighter extends SolrTestCaseJ4 {
+  
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+initCore("solrconfig-unifiedhighlight.xml", 
"schema-unifiedhighlight.xml");
+
+// test our config is sane, just to be sure:
+
+// postingshighlighter should be used
+SolrHighlighter highlighter = 
HighlightComponent.getHighlighter(h.getCore());
+assertTrue("wrong highlighter: " + highlighter.getClass(), highlighter 
instanceof UnifiedSolrHighlighter);
+
+// 'text' and 'text3' should have offsets, 'text2' should not
+IndexSchema schema = h.getCore().getLatestSchema();
+assertTrue(schema.getField("text").storeOffsetsWithPositions());
+assertTrue(schema.getField("text3").storeOffsetsWithPositions());
+assertFalse(schema.getField("text2").storeOffsetsWithPositions());
+  }
+  
+  @Override
+  public void setUp() throws Exception {
+super.setUp();
+clearIndex();
+assertU(adoc("text", "document one", "text2", "document one", "text3", 
"crappy document", "id", "101"));
+assertU(adoc("text", "second document", "text2", "second document", 
"text3", "crappier document", "id", "102"));
+assertU(commit());
+  }
+  
+  public void testSimple() {
+assertQ("simplest test", 
+req("q", "text:document", "sort", "id asc", "hl", "true"),
+"count(//lst[@name='highlighting']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/str='document
 one'",
+
"//lst[@name='highlighting']/lst[@name='102']/arr[@name='text']/str='second 
document'");
+  }
+
+  public void testMultipleSnippetsReturned() {
+clearIndex();
+assertU(adoc("text", "Document snippet one. Intermediate sentence. 
Document snippet two.",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("multiple snippets test",
+req("q", "text:document", "sort", "id asc", "hl", "true", 
"hl.snippets", "2", "hl.bs.type", "SENTENCE"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Document 
snippet one. '",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[2]='Document 
snippet two.'");
+  }
+
+  public void testStrictPhrasesEnabledByDefault() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be enabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=1",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Strict 
phrases should be enabled for phrases'");
+  }
+
+  public void testStrictPhrasesCanBeDisabled() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be disabled for phrases",
+

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662930#comment-15662930
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87738726
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java ---
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import java.io.IOException;
+import java.text.BreakIterator;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.postingshighlight.WholeBreakIterator;
+import org.apache.lucene.search.uhighlight.DefaultPassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageScorer;
+import org.apache.lucene.search.uhighlight.UnifiedHighlighter;
+import org.apache.solr.common.params.HighlightParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.core.PluginInfo;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.request.SolrRequestInfo;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.RTimerTree;
+import org.apache.solr.util.plugin.PluginInfoInitialized;
+
+/*
+ * TODO: The HighlightComponent should not call rewrite on the query; it 
should be up to the
+ * SolrHighlighter to do if needed.  Furthermore this arrangement is odd 
-- why are these abstractions separate?
+ */
+
+/**
+ * Highlighter impl that uses {@link UnifiedHighlighter}
+ * 
+ * Example configuration with default values:
+ * 
+ * requestHandler name="standard" 
class="solr.StandardRequestHandler"
+ * lst name="defaults"
+ * int name="hl.snippets"1/int
+ * str name="hl.tag.pre"lt;emgt;/str
+ * str name="hl.tag.post"lt;/emgt;/str
+ * str name="hl.tag.ellipsis"... /str
+ * bool name="hl.defaultSummary"true/bool
+ * str name="hl.encoder"simple/str
+ * float name="hl.score.k1"1.2/float
+ * float name="hl.score.b"0.75/float
+ * float name="hl.score.pivot"87/float
+ * str name="hl.bs.language"/str
+ * str name="hl.bs.country"/str
+ * str name="hl.bs.variant"/str
+ * str name="hl.bs.type"SENTENCE/str
+ * int name="hl.maxAnalyzedChars"1/int
+ * bool name="hl.highlightMultiTerm"true/bool
+ * /lst
+ * /requestHandler
+ * 
+ * ...
+ * 
+ * searchComponent class="solr.HighlightComponent" name="highlight"
+ * highlighting 
class="org.apache.solr.highlight.UnifiedSolrHighlighter"/
+ * /searchComponent
+ * 
+ * 
+ * Notes:
+ * 
+ * hl.q (string) can specify the query
+ * hl.fl (string) specifies the field list.
+ * hl.snippets (int) specifies how many snippets to return.
+ * hl.tag.pre (string) specifies text which appears before a 
highlighted term.
+ * hl.tag.post (string) specifies text which appears after a 
highlighted term.
+ * hl.tag.ellipsis (string) specifies text which joins non-adjacent 
passages. The default is to retain each
+ * value in a list without joining them.
+ * hl.defaultSummary (bool) specifies if a field should have a default 
summary of the leading text.
+ * hl.encoder (string) can be 'html' (html escapes content) or 
'simple' (no escaping).
+ * hl.score.k1 (float) specifies bm25 

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662929#comment-15662929
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740728
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
--- End diff --

obsolete reference to PostingsSolrHighlighter


> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662925#comment-15662925
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740749
  
--- Diff: 
solr/core/src/test-files/solr/collection1/conf/solrconfig-unifiedhighlight.xml 
---
@@ -0,0 +1,35 @@
+
+
+
+
+
--- End diff --

obsolete postings highlighter reference


> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662926#comment-15662926
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87743142
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java ---
@@ -373,6 +373,11 @@ protected BoundaryScanner getBoundaryScanner(String 
fieldName, SolrParams params
 if (!isHighlightingEnabled(params)) // also returns early if no unique 
key field
   return null;
 
+boolean rewrite = query != null && 
!(Boolean.valueOf(params.get(HighlightParams.USE_PHRASE_HIGHLIGHTER, "true")) &&
--- End diff --

no biggie but I think a simple if(...) condition would be simpler; no 
variable.


> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662928#comment-15662928
 ] 

ASF GitHub Bot commented on SOLR-9708:
--

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740550
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
+public class TestUnifiedSolrHighlighter extends SolrTestCaseJ4 {
+  
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+initCore("solrconfig-unifiedhighlight.xml", 
"schema-unifiedhighlight.xml");
+
+// test our config is sane, just to be sure:
+
+// postingshighlighter should be used
+SolrHighlighter highlighter = 
HighlightComponent.getHighlighter(h.getCore());
+assertTrue("wrong highlighter: " + highlighter.getClass(), highlighter 
instanceof UnifiedSolrHighlighter);
+
+// 'text' and 'text3' should have offsets, 'text2' should not
+IndexSchema schema = h.getCore().getLatestSchema();
+assertTrue(schema.getField("text").storeOffsetsWithPositions());
+assertTrue(schema.getField("text3").storeOffsetsWithPositions());
+assertFalse(schema.getField("text2").storeOffsetsWithPositions());
+  }
+  
+  @Override
+  public void setUp() throws Exception {
+super.setUp();
+clearIndex();
+assertU(adoc("text", "document one", "text2", "document one", "text3", 
"crappy document", "id", "101"));
+assertU(adoc("text", "second document", "text2", "second document", 
"text3", "crappier document", "id", "102"));
+assertU(commit());
+  }
+  
+  public void testSimple() {
+assertQ("simplest test", 
+req("q", "text:document", "sort", "id asc", "hl", "true"),
+"count(//lst[@name='highlighting']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/str='document
 one'",
+
"//lst[@name='highlighting']/lst[@name='102']/arr[@name='text']/str='second 
document'");
+  }
+
+  public void testMultipleSnippetsReturned() {
+clearIndex();
+assertU(adoc("text", "Document snippet one. Intermediate sentence. 
Document snippet two.",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("multiple snippets test",
+req("q", "text:document", "sort", "id asc", "hl", "true", 
"hl.snippets", "2", "hl.bs.type", "SENTENCE"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Document 
snippet one. '",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[2]='Document 
snippet two.'");
+  }
+
+  public void testStrictPhrasesEnabledByDefault() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be enabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=1",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Strict 
phrases should be enabled for phrases'");
+  }
+
+  public void testStrictPhrasesCanBeDisabled() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be disabled for phrases",
+

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87741032
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java ---
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import java.io.IOException;
+import java.text.BreakIterator;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.postingshighlight.WholeBreakIterator;
+import org.apache.lucene.search.uhighlight.DefaultPassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageScorer;
+import org.apache.lucene.search.uhighlight.UnifiedHighlighter;
+import org.apache.solr.common.params.HighlightParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.core.PluginInfo;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.request.SolrRequestInfo;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.RTimerTree;
+import org.apache.solr.util.plugin.PluginInfoInitialized;
+
+/*
+ * TODO: The HighlightComponent should not call rewrite on the query; it 
should be up to the
+ * SolrHighlighter to do if needed.  Furthermore this arrangement is odd 
-- why are these abstractions separate?
+ */
+
+/**
+ * Highlighter impl that uses {@link UnifiedHighlighter}
+ * 
+ * Example configuration with default values:
+ * 
+ * requestHandler name="standard" 
class="solr.StandardRequestHandler"
+ * lst name="defaults"
+ * int name="hl.snippets"1/int
+ * str name="hl.tag.pre"lt;emgt;/str
+ * str name="hl.tag.post"lt;/emgt;/str
+ * str name="hl.tag.ellipsis"... /str
+ * bool name="hl.defaultSummary"true/bool
+ * str name="hl.encoder"simple/str
+ * float name="hl.score.k1"1.2/float
+ * float name="hl.score.b"0.75/float
+ * float name="hl.score.pivot"87/float
+ * str name="hl.bs.language"/str
+ * str name="hl.bs.country"/str
+ * str name="hl.bs.variant"/str
+ * str name="hl.bs.type"SENTENCE/str
+ * int name="hl.maxAnalyzedChars"1/int
+ * bool name="hl.highlightMultiTerm"true/bool
+ * /lst
+ * /requestHandler
+ * 
+ * ...
+ * 
+ * searchComponent class="solr.HighlightComponent" name="highlight"
+ * highlighting 
class="org.apache.solr.highlight.UnifiedSolrHighlighter"/
+ * /searchComponent
+ * 
+ * 
+ * Notes:
+ * 
+ * hl.q (string) can specify the query
+ * hl.fl (string) specifies the field list.
+ * hl.snippets (int) specifies how many snippets to return.
+ * hl.tag.pre (string) specifies text which appears before a 
highlighted term.
+ * hl.tag.post (string) specifies text which appears after a 
highlighted term.
+ * hl.tag.ellipsis (string) specifies text which joins non-adjacent 
passages. The default is to retain each
+ * value in a list without joining them.
+ * hl.defaultSummary (bool) specifies if a field should have a default 
summary of the leading text.
+ * hl.encoder (string) can be 'html' (html escapes content) or 
'simple' (no escaping).
+ * hl.score.k1 (float) specifies bm25 scoring parameter 'k1'
+ * hl.score.b (float) specifies bm25 scoring parameter 'b'
+ * hl.score.pivot (float) specifies bm25 scoring parameter 'avgdl'
+ * hl.bs.type (string) specifies how to divide text into passages: 

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87738726
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/UnifiedSolrHighlighter.java ---
@@ -0,0 +1,366 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import java.io.IOException;
+import java.text.BreakIterator;
+import java.util.Collections;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Set;
+
+import org.apache.lucene.document.Document;
+import org.apache.lucene.search.DocIdSetIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.postingshighlight.WholeBreakIterator;
+import org.apache.lucene.search.uhighlight.DefaultPassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageFormatter;
+import org.apache.lucene.search.uhighlight.PassageScorer;
+import org.apache.lucene.search.uhighlight.UnifiedHighlighter;
+import org.apache.solr.common.params.HighlightParams;
+import org.apache.solr.common.params.SolrParams;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.SimpleOrderedMap;
+import org.apache.solr.core.PluginInfo;
+import org.apache.solr.request.SolrQueryRequest;
+import org.apache.solr.request.SolrRequestInfo;
+import org.apache.solr.schema.IndexSchema;
+import org.apache.solr.schema.SchemaField;
+import org.apache.solr.search.DocIterator;
+import org.apache.solr.search.DocList;
+import org.apache.solr.search.SolrIndexSearcher;
+import org.apache.solr.util.RTimerTree;
+import org.apache.solr.util.plugin.PluginInfoInitialized;
+
+/*
+ * TODO: The HighlightComponent should not call rewrite on the query; it 
should be up to the
+ * SolrHighlighter to do if needed.  Furthermore this arrangement is odd 
-- why are these abstractions separate?
+ */
+
+/**
+ * Highlighter impl that uses {@link UnifiedHighlighter}
+ * 
+ * Example configuration with default values:
+ * 
+ * requestHandler name="standard" 
class="solr.StandardRequestHandler"
+ * lst name="defaults"
+ * int name="hl.snippets"1/int
+ * str name="hl.tag.pre"lt;emgt;/str
+ * str name="hl.tag.post"lt;/emgt;/str
+ * str name="hl.tag.ellipsis"... /str
+ * bool name="hl.defaultSummary"true/bool
+ * str name="hl.encoder"simple/str
+ * float name="hl.score.k1"1.2/float
+ * float name="hl.score.b"0.75/float
+ * float name="hl.score.pivot"87/float
+ * str name="hl.bs.language"/str
+ * str name="hl.bs.country"/str
+ * str name="hl.bs.variant"/str
+ * str name="hl.bs.type"SENTENCE/str
+ * int name="hl.maxAnalyzedChars"1/int
+ * bool name="hl.highlightMultiTerm"true/bool
+ * /lst
+ * /requestHandler
+ * 
+ * ...
+ * 
+ * searchComponent class="solr.HighlightComponent" name="highlight"
+ * highlighting 
class="org.apache.solr.highlight.UnifiedSolrHighlighter"/
+ * /searchComponent
+ * 
+ * 
+ * Notes:
+ * 
+ * hl.q (string) can specify the query
+ * hl.fl (string) specifies the field list.
+ * hl.snippets (int) specifies how many snippets to return.
+ * hl.tag.pre (string) specifies text which appears before a 
highlighted term.
+ * hl.tag.post (string) specifies text which appears after a 
highlighted term.
+ * hl.tag.ellipsis (string) specifies text which joins non-adjacent 
passages. The default is to retain each
+ * value in a list without joining them.
+ * hl.defaultSummary (bool) specifies if a field should have a default 
summary of the leading text.
+ * hl.encoder (string) can be 'html' (html escapes content) or 
'simple' (no escaping).
+ * hl.score.k1 (float) specifies bm25 scoring parameter 'k1'
+ * hl.score.b (float) specifies bm25 scoring parameter 'b'
+ * hl.score.pivot (float) specifies bm25 scoring parameter 'avgdl'
+ * hl.bs.type (string) specifies how to divide text into passages: 

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740491
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
+public class TestUnifiedSolrHighlighter extends SolrTestCaseJ4 {
+  
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+initCore("solrconfig-unifiedhighlight.xml", 
"schema-unifiedhighlight.xml");
+
+// test our config is sane, just to be sure:
+
+// postingshighlighter should be used
+SolrHighlighter highlighter = 
HighlightComponent.getHighlighter(h.getCore());
+assertTrue("wrong highlighter: " + highlighter.getClass(), highlighter 
instanceof UnifiedSolrHighlighter);
+
+// 'text' and 'text3' should have offsets, 'text2' should not
+IndexSchema schema = h.getCore().getLatestSchema();
+assertTrue(schema.getField("text").storeOffsetsWithPositions());
+assertTrue(schema.getField("text3").storeOffsetsWithPositions());
+assertFalse(schema.getField("text2").storeOffsetsWithPositions());
+  }
+  
+  @Override
+  public void setUp() throws Exception {
+super.setUp();
+clearIndex();
+assertU(adoc("text", "document one", "text2", "document one", "text3", 
"crappy document", "id", "101"));
+assertU(adoc("text", "second document", "text2", "second document", 
"text3", "crappier document", "id", "102"));
+assertU(commit());
+  }
+  
+  public void testSimple() {
+assertQ("simplest test", 
+req("q", "text:document", "sort", "id asc", "hl", "true"),
+"count(//lst[@name='highlighting']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/str='document
 one'",
+
"//lst[@name='highlighting']/lst[@name='102']/arr[@name='text']/str='second 
document'");
+  }
+
+  public void testMultipleSnippetsReturned() {
+clearIndex();
+assertU(adoc("text", "Document snippet one. Intermediate sentence. 
Document snippet two.",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("multiple snippets test",
+req("q", "text:document", "sort", "id asc", "hl", "true", 
"hl.snippets", "2", "hl.bs.type", "SENTENCE"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Document 
snippet one. '",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[2]='Document 
snippet two.'");
+  }
+
+  public void testStrictPhrasesEnabledByDefault() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be enabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=1",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Strict 
phrases should be enabled for phrases'");
+  }
+
+  public void testStrictPhrasesCanBeDisabled() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be disabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true", 

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740728
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
--- End diff --

obsolete reference to PostingsSolrHighlighter


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740749
  
--- Diff: 
solr/core/src/test-files/solr/collection1/conf/solrconfig-unifiedhighlight.xml 
---
@@ -0,0 +1,35 @@
+
+
+
+
+
--- End diff --

obsolete postings highlighter reference


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87743142
  
--- Diff: 
solr/core/src/java/org/apache/solr/highlight/DefaultSolrHighlighter.java ---
@@ -373,6 +373,11 @@ protected BoundaryScanner getBoundaryScanner(String 
fieldName, SolrParams params
 if (!isHighlightingEnabled(params)) // also returns early if no unique 
key field
   return null;
 
+boolean rewrite = query != null && 
!(Boolean.valueOf(params.get(HighlightParams.USE_PHRASE_HIGHLIGHTER, "true")) &&
--- End diff --

no biggie but I think a simple if(...) condition would be simpler; no 
variable.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #107: SOLR-9708 UnifiedHighlighter Solr Plugin

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/107#discussion_r87740550
  
--- Diff: 
solr/core/src/test/org/apache/solr/highlight/TestUnifiedSolrHighlighter.java ---
@@ -0,0 +1,222 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.highlight;
+
+import org.apache.solr.SolrTestCaseJ4;
+import org.apache.solr.handler.component.HighlightComponent;
+import org.apache.solr.schema.IndexSchema;
+import org.junit.BeforeClass;
+import org.junit.Ignore;
+
+/** simple tests for PostingsSolrHighlighter */
+public class TestUnifiedSolrHighlighter extends SolrTestCaseJ4 {
+  
+  @BeforeClass
+  public static void beforeClass() throws Exception {
+initCore("solrconfig-unifiedhighlight.xml", 
"schema-unifiedhighlight.xml");
+
+// test our config is sane, just to be sure:
+
+// postingshighlighter should be used
+SolrHighlighter highlighter = 
HighlightComponent.getHighlighter(h.getCore());
+assertTrue("wrong highlighter: " + highlighter.getClass(), highlighter 
instanceof UnifiedSolrHighlighter);
+
+// 'text' and 'text3' should have offsets, 'text2' should not
+IndexSchema schema = h.getCore().getLatestSchema();
+assertTrue(schema.getField("text").storeOffsetsWithPositions());
+assertTrue(schema.getField("text3").storeOffsetsWithPositions());
+assertFalse(schema.getField("text2").storeOffsetsWithPositions());
+  }
+  
+  @Override
+  public void setUp() throws Exception {
+super.setUp();
+clearIndex();
+assertU(adoc("text", "document one", "text2", "document one", "text3", 
"crappy document", "id", "101"));
+assertU(adoc("text", "second document", "text2", "second document", 
"text3", "crappier document", "id", "102"));
+assertU(commit());
+  }
+  
+  public void testSimple() {
+assertQ("simplest test", 
+req("q", "text:document", "sort", "id asc", "hl", "true"),
+"count(//lst[@name='highlighting']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/str='document
 one'",
+
"//lst[@name='highlighting']/lst[@name='102']/arr[@name='text']/str='second 
document'");
+  }
+
+  public void testMultipleSnippetsReturned() {
+clearIndex();
+assertU(adoc("text", "Document snippet one. Intermediate sentence. 
Document snippet two.",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("multiple snippets test",
+req("q", "text:document", "sort", "id asc", "hl", "true", 
"hl.snippets", "2", "hl.bs.type", "SENTENCE"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=2",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Document 
snippet one. '",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[2]='Document 
snippet two.'");
+  }
+
+  public void testStrictPhrasesEnabledByDefault() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be enabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true"),
+
"count(//lst[@name='highlighting']/lst[@name='101']/arr[@name='text']/*)=1",
+
"//lst[@name='highlighting']/lst[@name='101']/arr/str[1]='Strict 
phrases should be enabled for phrases'");
+  }
+
+  public void testStrictPhrasesCanBeDisabled() {
+clearIndex();
+assertU(adoc("text", "Strict phrases should be disabled for phrases",
+"text2", "document one", "text3", "crappy document", "id", "101"));
+assertU(commit());
+assertQ("strict phrase handling",
+req("q", "text:\"strict phrases\"", "sort", "id asc", "hl", 
"true", 

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_102) - Build # 18278 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18278/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.handler.TestSolrConfigHandlerCloud.test

Error Message:


Stack Trace:
java.lang.NullPointerException
at 
__randomizedtesting.SeedInfo.seed([D0DD9E435422B15C:5889A199FADEDCA4]:0)
at 
org.apache.solr.core.TestSolrConfigHandler.reqhandlertests(TestSolrConfigHandler.java:454)
at 
org.apache.solr.handler.TestSolrConfigHandlerCloud.testReqHandlerAPIs(TestSolrConfigHandlerCloud.java:97)
at 
org.apache.solr.handler.TestSolrConfigHandlerCloud.test(TestSolrConfigHandlerCloud.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:985)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Updated] (LUCENE-7559) UnifiedHighlighter: Make Passage public for extensibility

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated LUCENE-7559:
-
Attachment: LUCENE-7559.patch

This patch makes it's fields private, adds setters, and makes the package 
visible methods public.  {{/** @lucene.internal */}} was added to methods only 
needed internally.

> UnifiedHighlighter: Make Passage public for extensibility
> -
>
> Key: LUCENE-7559
> URL: https://issues.apache.org/jira/browse/LUCENE-7559
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/highlighter
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Attachments: LUCENE-7559.patch
>
>
> (UnifiedHighlighter) Passage.java has a variety of package-level fields and 
> methods that are accessed by FieldHighlighter.highlightOffsetsEnums (the 
> kernel of the UH).  It should be possible for people to extend this method to 
> tweak the process of producing passages.  That method is extensible but 
> Passage.java is too closed.  Even if one were to write their own equivalent 
> to Passage.java, it would be impossible to then use PassageScorer or 
> PassageFormatter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-7559) UnifiedHighlighter: Make Passage public for extensibility

[David Smiley (JIRA)]

David Smiley created LUCENE-7559:


 Summary: UnifiedHighlighter: Make Passage public for extensibility
 Key: LUCENE-7559
 URL: https://issues.apache.org/jira/browse/LUCENE-7559
 Project: Lucene - Core
  Issue Type: Improvement
  Components: modules/highlighter
Reporter: David Smiley
Assignee: David Smiley
Priority: Minor


(UnifiedHighlighter) Passage.java has a variety of package-level fields and 
methods that are accessed by FieldHighlighter.highlightOffsetsEnums (the kernel 
of the UH).  It should be possible for people to extend this method to tweak 
the process of producing passages.  That method is extensible but Passage.java 
is too closed.  Even if one were to write their own equivalent to Passage.java, 
it would be impossible to then use PassageScorer or PassageFormatter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7526) Improvements to UnifiedHighlighter OffsetStrategies

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662505#comment-15662505
 ] 

David Smiley commented on LUCENE-7526:
--

RE Benchmarks:

Yeah it seems hard to get consistency. I even upped the iterations counts to 
2000 (from 500) and saw more variability than I'd like.  Nonetheless your 
analysis looks good to me; it's how I would sum up my observations today.

I ran my benchmarks with "-Xms4G -Xmx4G -XX:NewRatio=1" (to reduce the effects 
of GC).  My machine is a MacBook Pro Retina "Late 2013", 2 GHz i7, 8GB RAM.  
This index is on an external spinning disk but there is no disk activity after 
warm-up because it's all in the O.S. Cache.

As I looked over things carefully, I made a few more changes; all pretty minor. 
 A few were in reaction to "ant precommit". 
https://github.com/dsmiley/lucene-solr/commits/uh_Tim -- the test & some 
javadoc issues.  I also did a little bit of refactoring I hope you'll be good 
with.  In a couple cases I merely moved a method up or down so that the code 
flows top to bottom better.  Regarding Passage: I'll create a separate issue 
expressly for making Passage usable by anyone customizing the highlighter.  
It's not as simple as addMatch being public; there are other methods.  

_What's there now is in good shape for committing._

A couple ideas occurred to me; _feel free to punt to another issue or never_:
* MultiValueTokenStream isn't needed for MemoryIndexOffsetStrategy, albeit with 
a change to loop over content.split(separatorChar).  MemoryIndex.addField is 
overloaded to take the position increment gap.  Then, MultiValueTokenStream 
could move to an inner class of TokenStreamOffsetStrategy, and it wouldn't 
generally be used (as it's no longer by default).  That'd be nice -- keeping 
the complexity over there, and it's a bit of a hack too.
* We had made OffsetsEnum & TokenStreamPostingsEnum implement Closeable to 
ameliorate the ramifications of the text analysis code throwing an exception, 
i.e. due to a bug.  The only beneficiary of this now is 
TokenStreamOffsetStrategy, which isn't the default anymore. It could be removed 
to simplify things.  But then again, perhaps it could be useful for those 
implementing custom OffsetStrategies.  I guess it should stay; there's very 
little to this after all.

Proposed CHANGES.txt in "Improvements":
* Enhanced UnifiedHighlighter's passage relevancy for queries with wildcards 
and sometimes just terms. Added shouldPreferPassageRelevancyOverSpeed() which 
can be overridden to return false to eek out more speed in some cases. (Tim 
Rodriguez, David Smiley)

> Improvements to UnifiedHighlighter OffsetStrategies
> ---
>
> Key: LUCENE-7526
> URL: https://issues.apache.org/jira/browse/LUCENE-7526
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
>Priority: Minor
> Fix For: 6.4
>
>
> This ticket improves several of the UnifiedHighlighter FieldOffsetStrategies 
> by reducing reliance on creating or re-creating TokenStreams.
> The primary changes are as follows:
> * AnalysisOffsetStrategy - split into two offset strategies
>   ** MemoryIndexOffsetStrategy - the primary analysis mode that utilizes a 
> MemoryIndex for producing Offsets
>   ** TokenStreamOffsetStrategy - an offset strategy that avoids creating a 
> MemoryIndex.  Can only be used if the query distills down to terms and 
> automata.
> * TokenStream removal 
>   ** MemoryIndexOffsetStrategy - previously a TokenStream was created to fill 
> the memory index and then once consumed a new one was generated by 
> uninverting the MemoryIndex back into a TokenStream if there were automata 
> (wildcard/mtq queries) involved.  Now this is avoided, which should save 
> memory and avoid a second pass over the data.
>   ** TermVectorOffsetStrategy - this was refactored in a similar way to avoid 
> generating a TokenStream if automata are involved.
>   ** PostingsWithTermVectorsOffsetStrategy - similar refactoring
> * CompositePostingsEnum - aggregates several underlying PostingsEnums for 
> wildcard/mtq queries.  This should improve relevancy by providing unified 
> metrics for a wildcard across all it's term matches
> * Added a HighlightFlag for enabling the newly separated 
> TokenStreamOffsetStrategy since it can adversely affect passage relevancy



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.x - Build # 198 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.x/198/

4 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testResilienceWithDeleteByQueryOnTarget

Error Message:
Timeout while trying to assert number of documents @ target_collection

Stack Trace:
java.lang.AssertionError: Timeout while trying to assert number of documents @ 
target_collection
at 
__randomizedtesting.SeedInfo.seed([B6A88B4914AF140A:16CA19D3C4F6D14F]:0)
at 
org.apache.solr.cloud.BaseCdcrDistributedZkTest.assertNumDocs(BaseCdcrDistributedZkTest.java:271)
at 
org.apache.solr.cloud.CdcrReplicationDistributedZkTest.testResilienceWithDeleteByQueryOnTarget(CdcrReplicationDistributedZkTest.java:595)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Cao Manh Dat (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662327#comment-15662327
 ] 

Cao Manh Dat commented on SOLR-8593:


Thanks [~julianhyde], that make sense, but will Calcite have an option for that 
in the future? Because along with JDBC API, we also support query through REST 
API.

For example 
{code}
curl --data-urlencode 'stmt=SELECT to, count(*) FROM collection4 GROUP BY to 
ORDER BY count(*) desc LIMIT 10' 
  http://localhost:8983/solr/collection4/sql?aggregationMode=facet
{code}

Below is sample result set:
{code}
{"result-set":{"docs":[
  {"count(*)":9158,"to":"pete.da...@enron.com"},
  {"count(*)":6244,"to":"tana.jo...@enron.com"},
  {"count(*)":5874,"to":"jeff.dasov...@enron.com"},
  {"count(*)":5867,"to":"sara.shackle...@enron.com"},
  {"count(*)":5595,"to":"steven.k...@enron.com"},
  {"count(*)":4904,"to":"vkamin...@aol.com"},
  {"count(*)":4622,"to":"mark.tay...@enron.com"},
  {"count(*)":3819,"to":"kay.m...@enron.com"},
  {"count(*)":3678,"to":"richard.shap...@enron.com"},
  {"count(*)":3653,"to":"kate.sy...@enron.com"},
  {"EOF":"true","RESPONSE_TIME":10}]}
}
{code}
So "EXPR$1" is kinda weird as a field name.

I also want to ask another question, right now we rely on Calcite in memory 
filter to filter expr in having clause ( as {{ having sum(field_i) = 19 }} in 
below query
{code}
select str_s, count(*) as myCount, sum(field_i), min(field_i), max(field_i), 
cast(avg(1.0 * field_i) as float) from collection1 where text='' group by 
str_s having sum(field_i) = 19 order by sum(field_i) asc
{code}

So I created a filter predicate for convert LogicalFilter to SolrFilter like 
this
{code}
private static final boolean isNotFilterByExpr (List rexNodes, 
List fieldNames) {
  // We dont have a way to filter by result of aggregator now
  boolean result = true;
  for (RexNode rexNode : rexNodes) {
  if (rexNode instanceof RexCall) {
  result = result && isNotFilterByExpr(((RexCall) rexNode).getOperands(), 
fieldNames);
  } else if (rexNode instanceof RexInputRef) {
  result = result && !fieldNames.get(((RexInputRef) 
rexNode).getIndex()).startsWith("EXPR$");
  }
  }
  return result;
}

private static final Predicate FILTER_PREDICATE = relNode -> {
  List filterOperands = ((RexCall) ((LogicalFilter) 
relNode).getCondition()).getOperands();
  return isNotFilterByExpr(filterOperands, 
SolrRules.solrFieldNames(relNode.getRowType()));
};
{code}

But the above code is not works when I add alias for sum(field_i) like this
{code}
select str_s, count(*) as myCount, sum(field_i) as mySum, min(field_i), 
max(field_i), cast(avg(1.0 * field_i) as float) from collection1 where 
text='' group by str_s having sum(field_i) = 19 order by sum(field_i) asc
{code}

So do Calcite have a way to check for RexInputRef is an alias or not?


> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8593) Integrate Apache Calcite into the SQLHandler

[Julian Hyde (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662280#comment-15662280
 ] 

Julian Hyde commented on SOLR-8593:
---

"count(\*)" is not a good derived column name, because it contains 
non-alphanumeric characters and is therefore not a valid identifier unless you 
enclose it in double-quotes. Therefore Calcite generates an alias that is a 
valid identifier.

I believe quite a few other databases do this.

> Integrate Apache Calcite into the SQLHandler
> 
>
> Key: SOLR-8593
> URL: https://issues.apache.org/jira/browse/SOLR-8593
> Project: Solr
>  Issue Type: Improvement
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
> Attachments: SOLR-8593.patch
>
>
>The Presto SQL Parser was perfect for phase one of the SQLHandler. It was 
> nicely split off from the larger Presto project and it did everything that 
> was needed for the initial implementation.
> Phase two of the SQL work though will require an optimizer. Here is where 
> Apache Calcite comes into play. It has a battle tested cost based optimizer 
> and has been integrated into Apache Drill and Hive.
> This work can begin in trunk following the 6.0 release. The final query plans 
> will continue to be translated to Streaming API objects (TupleStreams), so 
> continued work on the JDBC driver should plug in nicely with the Calcite work.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: lucene-solr:master: LUCENE-7555: use BM25Similarity for this test

[Michael McCandless]

OK thanks Rob, I will revert my changes (that made test coverage of
our similarities wimpier) and pull BooleanSimilarity from the random
rotation instead.

Mike McCandless

http://blog.mikemccandless.com

On Sun, Nov 13, 2016 at 9:16 AM, Robert Muir  wrote:
> By the way, the third option, even better, is to replace all the stuff
> these tests are doing with direct ones in the similarity unit tests.
> For example TestSimilarityBase.testCrazyIndexTimeBoosts()
>
> But that is a good deal of work. Without that, its important for
> tests, somewhere, somehow to catch when these scoring systems are not
> well-behaved (e.g. some bug in the formula causing scoring formula to
> go backwards and many other things).
>
> That is why not all Similarities are in RandomSImilarity, because some
> still cannot meet that bar and have crazy behavior, some may be
> unfixable! But the tests found tons of bugs in these formulas, where
> bad things happen when their expected distribution is not met, or
> where features of lucene like boost don't play well with them. It is
> important to have tests, somewhere, somehow, that will find these
> things.
>
> Making all the tests wimpy so that BooleanSimilarity can be in there
> doesn't make any sense to me.
>
>
> On Sun, Nov 13, 2016 at 8:58 AM, Robert Muir  wrote:
>> I think these fixes are not the best way to go.
>>
>> Overall, these commits destroy large features of lucene's test suite,
>> for testing that similarities are well-behaved and disables them.
>> Previously, they were randomized across all similarities: This is the
>> whole point of doing that.
>>
>> But why add BooleanSimilarity to the random list at all, when any test
>> failures it causes (because of its own brokenness!), will be disabled
>> to use only BM25 This hurts all of lucene.
>>
>> Instead, I think all these test changes should be reverted and either:
>> 1) BooleanSimilarity should be fixed to not be so brain-dead: e.g. why
>> does it ignore index-time boost but not query-time boost? This is
>> buggy are some of the reasons these tests fail.
>>
>> or
>>
>> 2) Remove BooleanSimilarity from the randomized testing setup here.
>>
>>
>>
>>
>> On Sat, Nov 12, 2016 at 1:29 PM,   wrote:
>>> Repository: lucene-solr
>>> Updated Branches:
>>>   refs/heads/master 4d9451034 -> 2bc1d2761
>>>
>>>
>>> LUCENE-7555: use BM25Similarity for this test
>>>
>>>
>>> Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
>>> Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2bc1d276
>>> Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2bc1d276
>>> Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2bc1d276
>>>
>>> Branch: refs/heads/master
>>> Commit: 2bc1d2761f593b2ceea19b3195e0cb430318ceaa
>>> Parents: 4d94510
>>> Author: Mike McCandless 
>>> Authored: Sat Nov 12 13:28:57 2016 -0500
>>> Committer: Mike McCandless 
>>> Committed: Sat Nov 12 13:28:57 2016 -0500
>>>
>>> --
>>>  .../src/test/org/apache/solr/uninverting/TestFieldCacheSort.java | 4 +++-
>>>  1 file changed, 3 insertions(+), 1 deletion(-)
>>> --
>>>
>>>
>>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2bc1d276/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>>> --
>>> diff --git 
>>> a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java 
>>> b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>>> index d53f610..4755c8f 100644
>>> --- a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>>> +++ b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>>> @@ -50,10 +50,11 @@ import org.apache.lucene.search.Sort;
>>>  import org.apache.lucene.search.SortField;
>>>  import org.apache.lucene.search.TermQuery;
>>>  import org.apache.lucene.search.TopDocs;
>>> +import org.apache.lucene.search.similarities.BM25Similarity;
>>>  import org.apache.lucene.store.Directory;
>>> -import org.apache.solr.uninverting.UninvertingReader.Type;
>>>  import org.apache.lucene.util.LuceneTestCase;
>>>  import org.apache.lucene.util.TestUtil;
>>> +import org.apache.solr.uninverting.UninvertingReader.Type;
>>>
>>>  /*
>>>   * Tests sorting (but with fieldcache instead of docvalues)
>>> @@ -434,6 +435,7 @@ public class TestFieldCacheSort extends LuceneTestCase {
>>>  writer.close();
>>>
>>>  IndexSearcher searcher = newSearcher(ir);
>>> +searcher.setSimilarity(new BM25Similarity());
>>>  Sort sort = new Sort(new SortField(null, SortField.Type.SCORE, true));
>>>
>>>  TopDocs actual = searcher.search(new TermQuery(new Term("value", 
>>> "foo")), 10, sort);
>>>
>
> 

[jira] [Commented] (SOLR-9757) TestFieldCacheSort.testFieldScoreReverse() failure

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662170#comment-15662170
 ] 

Michael McCandless commented on SOLR-9757:
--

Woops, thanks [~steve_rowe] ... I forgot that 6.x also had the Solr version of 
this test.

> TestFieldCacheSort.testFieldScoreReverse() failure
> --
>
> Key: SOLR-9757
> URL: https://issues.apache.org/jira/browse/SOLR-9757
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Steve Rowe
>Assignee: Steve Rowe
>
> My Jenkins found a reproducing seed on branch_6x:
> {noformat}
>   [junit4] Suite: org.apache.solr.uninverting.TestFieldCacheSort
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFieldCacheSort 
> -Dtests.method=testFieldScoreReverse -Dtests.seed=7A20300F80199575 
> -Dtests.slow=true -Dtests.locale=th -Dtests.timezone=Chile/Continental 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>   [junit4] FAILURE 0.04s J1  | TestFieldCacheSort.testFieldScoreReverse <<<
>   [junit4]> Throwable #1: java.lang.AssertionError: expected:<0> but 
> was:<1>
>   [junit4]>   at 
> __randomizedtesting.SeedInfo.seed([7A20300F80199575:1E7CB3871D41FF6]:0)
>   [junit4]>   at 
> org.apache.solr.uninverting.TestFieldCacheSort.testFieldScoreReverse(TestFieldCacheSort.java:445)
>   [junit4]>   at java.lang.Thread.run(Thread.java:745)
>   [junit4]   2> NOTE: test params are: codec=Lucene62, 
> sim=RandomSimilarity(queryNorm=false,coord=crazy): {contents=DFR 
> I(n)L3(800.0), body=BM25(k1=1.2,b=0.75), 
> value=org.apache.lucene.search.similarities.BooleanSimilarity@127b0929}, 
> locale=th, timezone=Chile/Continental
>   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=294612296,total=529006592
>   [junit4]   2> NOTE: All tests run in this JVM: [BaseCdcrDistributedZkTest, 
> HdfsTlogReplayBufferedWhileIndexingTest, TestBlobHandler, 
> TestFreeTextSuggestions, ShardSplitTest, TestSolrIndexConfig, TestLockTree, 
> TestFiltering, StatsComponentTest, PeerSyncWithIndexFingerprintCachingTest, 
> TestRTimerTree, TestConfig, TestSubQueryTransformerDistrib, 
> TestStandardQParsers, LeaderElectionContextKeyTest, 
> TestUseDocValuesAsStored2, DebugComponentTest, UUIDFieldTest, TestRecovery, 
> CollectionReloadTest, TestFieldCacheSort]
>   [junit4] Completed [319/651 (2!)] on J1 in 1.52s, 60 tests, 1 failure <<< 
> FAILURES!
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Windows (64bit/jdk1.8.0_102) - Build # 559 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/559/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure

Error Message:
The data directory was not cleaned up on unload after a failed core reload

Stack Trace:
java.lang.AssertionError: The data directory was not cleaned up on unload after 
a failed core reload
at 
__randomizedtesting.SeedInfo.seed([D6C32CA016533463:AD0A8EEC3571E662]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure(CoreAdminHandlerTest.java:334)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11887 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.CoreAdminHandlerTest
   [junit4]   2> Creating 

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[Timothy M. Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662087#comment-15662087
 ] 

Timothy M. Rodriguez commented on SOLR-9708:


Let me know what you think.  If it looks good, I think we can commit it.

> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9708) Expose UnifiedHighlighter in Solr

[Timothy M. Rodriguez (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662086#comment-15662086
 ] 

Timothy M. Rodriguez commented on SOLR-9708:


I've pushed tests for the configurable items in the UH as well as for support 
of multiple snippets.  In addition a change was done to push highlighter 
specific logic down into the DefaultSolrHighlighter that was in the 
HighlightComponent (thanks [~dsmiley] for pointing that out).

> Expose UnifiedHighlighter in Solr
> -
>
> Key: SOLR-9708
> URL: https://issues.apache.org/jira/browse/SOLR-9708
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: highlighter
>Reporter: Timothy M. Rodriguez
>Assignee: David Smiley
> Fix For: 6.4
>
>
> This ticket is for creating a Solr plugin that can utilize the new 
> UnifiedHighlighter which was initially committed in 
> https://issues.apache.org/jira/browse/LUCENE-7438



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7466) Allow optional leading wildcards in complexphrase

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662067#comment-15662067
 ] 

Mikhail Khludnev commented on SOLR-7466:


I tried to hook up {{ReversedWildcardFilterFactory}} SOLR-1321 in 
\{!complexphrase}. The problem is that 
{code}
 ComplexPhraseQueryParser --|> o.a.lucene.q.c.QueryParser
{code}
But a cool code, which leverages {{ReversedWildcardFilterFactory}} resides in 
{{o.a.solr...SolrQueryParserBase.getWildcardQuery(String, String)}}.
So far, I copy-pasted CPQP to make it descendant of Solr's parser. It works, 
but it's not cute. Is there a better idea? Is it really necessary to deal with 
RWFF?
 


> Allow optional leading wildcards in complexphrase
> -
>
> Key: SOLR-7466
> URL: https://issues.apache.org/jira/browse/SOLR-7466
> Project: Solr
>  Issue Type: Improvement
>  Components: query parsers
>Affects Versions: 4.8
>Reporter: Andy hardin
>  Labels: complexPhrase, query-parser, wildcards
>
> Currently ComplexPhraseQParser (SOLR-1604) allows trailing wildcards on terms 
> in a phrase, but does not allow leading wildcards.  I would like the option 
> to be able to search for terms with both trailing and leading wildcards.  
> For example with:
> {!complexphrase allowLeadingWildcard=true} "j* *th"
> would match "John Smith", "Jim Smith", but not "John Schmitt"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-629) Fuzzy search with DisMax request handler

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15662035#comment-15662035
 ] 

Mikhail Khludnev commented on SOLR-629:
---

It would be great to support full query syntax for {{qf}} clauses, eg when 
parent child fields are searched it needs to support \{!parent}, \{!child}. Any 
ideas?  

> Fuzzy search with DisMax request handler
> 
>
> Key: SOLR-629
> URL: https://issues.apache.org/jira/browse/SOLR-629
> Project: Solr
>  Issue Type: Improvement
>  Components: query parsers
>Affects Versions: 1.3
>Reporter: Guillaume Smet
>Priority: Minor
> Attachments: SOLR-629.patch, SOLR-629.patch, 
> dismax_fuzzy_query_field.v0.1.diff, dismax_fuzzy_query_field.v0.1.diff
>
>
> The DisMax search handler doesn't support fuzzy queries which would be quite 
> useful for our usage of Solr and from what I've seen on the list, it's 
> something several people would like to have.
> Following this discussion 
> http://markmail.org/message/tx6kqr7ga6ponefa#query:solr%20dismax%20fuzzy+page:1+mid:c4pciq6rlr4dwtgm+state:results
>  , I added the ability to add fuzzy query field in the qf parameter. I kept 
> the patch as conservative as possible.
> The syntax supported is: fieldOne^2.3 fieldTwo~0.3 fieldThree~0.2^-0.4 
> fieldFour as discussed in the above thread.
> The recursive query aliasing should work even with fuzzy query fields using a 
> very simple rule: the aliased fields inherit the minSimilarity of their 
> parent, combined with their own one if they have one.
> Only the qf parameter support this syntax atm. I suppose we should make it 
> usable in pf too. Any opinion?
> Comments are very welcome, I'll spend the time needed to put this patch in 
> good shape.
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-7534) smokeTestRelease.py on cygwin [Errno 2] No such file or directory:

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev resolved LUCENE-7534.
--
Resolution: Fixed

> smokeTestRelease.py on cygwin [Errno 2] No such file or directory: 
> ---
>
> Key: LUCENE-7534
> URL: https://issues.apache.org/jira/browse/LUCENE-7534
> Project: Lucene - Core
>  Issue Type: Bug
> Environment: Windows, Cygwin, Python
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
> Fix For: master (7.0), 6.4
>
> Attachments: 6.3.0 RC3 smoke passed on cygwin.txt, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, cloud test fail on 
> smoke test.log, fail due to long classpath test.log
>
>
> {quote}
> $ python3 -u dev-tools/scripts/smokeTestRelease.py 
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC2-rev1fe1a54db32b8c27bfae81887cd4d75242090613/
> Revision: 1fe1a54db32b8c27bfae81887cd4d75242090613
> Java 1.8 JAVA_HOME=C:\Program Files\Java\jdk1.8.0_102
> Traceback (most recent call last):
>   File "dev-tools/scripts/smokeTestRelease.py", line 1440, in 
> main()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1377, in main
> c = parse_config()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1239, in parse_config
> c.java = make_java_config(parser, c.test_java8)
>   File "dev-tools/scripts/smokeTestRelease.py", line 1193, in make_java_config
> run_java8 = _make_runner(java8_home, '1.8')
>   File "dev-tools/scripts/smokeTestRelease.py", line 1179, in _make_runner
> java_home = subprocess.check_output('cygpath -u "%s"' % 
> java_home).read().decode('utf-8').strip()
>   File "/usr/lib/python3.4/subprocess.py", line 607, in check_output
> with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
>   File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
> restore_signals, start_new_session)
>   File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
> raise child_exception_type(errno_num, err_msg)
> FileNotFoundError: [Errno 2] No such file or directory: 'cygpath -u 
> "C:\\Program Files\\Java\\jdk1.8.0_102"'
> {quote}
> giving [the doc|https://docs.python.org/3.4/library/subprocess.html] path and 
> args should be either supplied as array of terms or supplied as {{shell=True}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-7534) smokeTestRelease.py on cygwin [Errno 2] No such file or directory:

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev reassigned LUCENE-7534:


Assignee: Mikhail Khludnev

> smokeTestRelease.py on cygwin [Errno 2] No such file or directory: 
> ---
>
> Key: LUCENE-7534
> URL: https://issues.apache.org/jira/browse/LUCENE-7534
> Project: Lucene - Core
>  Issue Type: Bug
> Environment: Windows, Cygwin, Python
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
> Fix For: master (7.0), 6.4
>
> Attachments: 6.3.0 RC3 smoke passed on cygwin.txt, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, cloud test fail on 
> smoke test.log, fail due to long classpath test.log
>
>
> {quote}
> $ python3 -u dev-tools/scripts/smokeTestRelease.py 
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC2-rev1fe1a54db32b8c27bfae81887cd4d75242090613/
> Revision: 1fe1a54db32b8c27bfae81887cd4d75242090613
> Java 1.8 JAVA_HOME=C:\Program Files\Java\jdk1.8.0_102
> Traceback (most recent call last):
>   File "dev-tools/scripts/smokeTestRelease.py", line 1440, in 
> main()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1377, in main
> c = parse_config()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1239, in parse_config
> c.java = make_java_config(parser, c.test_java8)
>   File "dev-tools/scripts/smokeTestRelease.py", line 1193, in make_java_config
> run_java8 = _make_runner(java8_home, '1.8')
>   File "dev-tools/scripts/smokeTestRelease.py", line 1179, in _make_runner
> java_home = subprocess.check_output('cygpath -u "%s"' % 
> java_home).read().decode('utf-8').strip()
>   File "/usr/lib/python3.4/subprocess.py", line 607, in check_output
> with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
>   File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
> restore_signals, start_new_session)
>   File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
> raise child_exception_type(errno_num, err_msg)
> FileNotFoundError: [Errno 2] No such file or directory: 'cygpath -u 
> "C:\\Program Files\\Java\\jdk1.8.0_102"'
> {quote}
> giving [the doc|https://docs.python.org/3.4/library/subprocess.html] path and 
> args should be either supplied as array of terms or supplied as {{shell=True}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-7534) smokeTestRelease.py on cygwin [Errno 2] No such file or directory:

[Mikhail Khludnev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mikhail Khludnev updated LUCENE-7534:
-
Fix Version/s: 6.4
   master (7.0)

> smokeTestRelease.py on cygwin [Errno 2] No such file or directory: 
> ---
>
> Key: LUCENE-7534
> URL: https://issues.apache.org/jira/browse/LUCENE-7534
> Project: Lucene - Core
>  Issue Type: Bug
> Environment: Windows, Cygwin, Python
>Reporter: Mikhail Khludnev
>Assignee: Mikhail Khludnev
> Fix For: master (7.0), 6.4
>
> Attachments: 6.3.0 RC3 smoke passed on cygwin.txt, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, 
> LUCENE-7534.patch, LUCENE-7534.patch, LUCENE-7534.patch, cloud test fail on 
> smoke test.log, fail due to long classpath test.log
>
>
> {quote}
> $ python3 -u dev-tools/scripts/smokeTestRelease.py 
> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.3.0-RC2-rev1fe1a54db32b8c27bfae81887cd4d75242090613/
> Revision: 1fe1a54db32b8c27bfae81887cd4d75242090613
> Java 1.8 JAVA_HOME=C:\Program Files\Java\jdk1.8.0_102
> Traceback (most recent call last):
>   File "dev-tools/scripts/smokeTestRelease.py", line 1440, in 
> main()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1377, in main
> c = parse_config()
>   File "dev-tools/scripts/smokeTestRelease.py", line 1239, in parse_config
> c.java = make_java_config(parser, c.test_java8)
>   File "dev-tools/scripts/smokeTestRelease.py", line 1193, in make_java_config
> run_java8 = _make_runner(java8_home, '1.8')
>   File "dev-tools/scripts/smokeTestRelease.py", line 1179, in _make_runner
> java_home = subprocess.check_output('cygpath -u "%s"' % 
> java_home).read().decode('utf-8').strip()
>   File "/usr/lib/python3.4/subprocess.py", line 607, in check_output
> with Popen(*popenargs, stdout=PIPE, **kwargs) as process:
>   File "/usr/lib/python3.4/subprocess.py", line 859, in __init__
> restore_signals, start_new_session)
>   File "/usr/lib/python3.4/subprocess.py", line 1457, in _execute_child
> raise child_exception_type(errno_num, err_msg)
> FileNotFoundError: [Errno 2] No such file or directory: 'cygpath -u 
> "C:\\Program Files\\Java\\jdk1.8.0_102"'
> {quote}
> giving [the doc|https://docs.python.org/3.4/library/subprocess.html] path and 
> args should be either supplied as array of terms or supplied as {{shell=True}}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6223 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6223/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure

Error Message:
The data directory was not cleaned up on unload after a failed core reload

Stack Trace:
java.lang.AssertionError: The data directory was not cleaned up on unload after 
a failed core reload
at 
__randomizedtesting.SeedInfo.seed([752F8C965512637F:EE62EDA7630B17E]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure(CoreAdminHandlerTest.java:334)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11160 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.CoreAdminHandlerTest
   [junit4]   2> Creating 

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1152 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1152/

7 tests failed.
FAILED:  org.apache.lucene.search.TestFuzzyQuery.testRandom

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([5A1ECDBBC9CEA9EF]:0)


FAILED:  junit.framework.TestSuite.org.apache.lucene.search.TestFuzzyQuery

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([5A1ECDBBC9CEA9EF]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest

Error Message:
ObjectTracker found 10 object(s) that were not released!!! [InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient, InternalHttpClient, InternalHttpClient, InternalHttpClient, 
InternalHttpClient] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:43)
  at 
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:267)
  at 
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:214)
  at 
org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:201)
  at 
org.apache.solr.client.solrj.impl.HttpSolrClient.(HttpSolrClient.java:210)
  at 
org.apache.solr.client.solrj.impl.HttpSolrClient.(HttpSolrClient.java:183)
  at 
org.apache.solr.client.solrj.impl.HttpSolrClient.(HttpSolrClient.java:167)
  at org.apache.solr.SolrTestCaseJ4.getHttpSolrClient(SolrTestCaseJ4.java:2250) 
 at 
org.apache.solr.cloud.ConcurrentDeleteAndCreateCollectionTest.testConcurrentCreateAndDeleteDoesNotFail(ConcurrentDeleteAndCreateCollectionTest.java:62)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)  at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)  
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:498)  at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
  at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
  at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
  at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
  at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
  at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
  at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
  at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
  at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
  at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
  at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
  at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
  at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
  at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at 

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+140) - Build # 18274 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/18274/
Java: 64bit/jdk-9-ea+140 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication

Error Message:
expected:<1> but was:<0>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<0>
at 
__randomizedtesting.SeedInfo.seed([286FF5509FFDCA9C:DF1C1B085915657A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.handler.TestReplicationHandler.doTestIndexAndConfigAliasReplication(TestReplicationHandler.java:1329)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:535)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 

[JENKINS] Lucene-Solr-6.x-Solaris (64bit/jdk1.8.0) - Build # 490 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Solaris/490/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
shard1 is not consistent.  Got 355 from http://127.0.0.1:35604/collection1 
(previous client) and got 356 from http://127.0.0.1:49884/collection1

Stack Trace:
java.lang.AssertionError: shard1 is not consistent.  Got 355 from 
http://127.0.0.1:35604/collection1 (previous client) and got 356 from 
http://127.0.0.1:49884/collection1
at 
__randomizedtesting.SeedInfo.seed([3E8638C1B1833781:B6D2071B1F7F5A79]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.checkShardConsistency(AbstractFullDistribZkTestBase.java:1271)
at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:228)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 

Re: lucene-solr:master: LUCENE-7555: use BM25Similarity for this test

[Robert Muir]

By the way, the third option, even better, is to replace all the stuff
these tests are doing with direct ones in the similarity unit tests.
For example TestSimilarityBase.testCrazyIndexTimeBoosts()

But that is a good deal of work. Without that, its important for
tests, somewhere, somehow to catch when these scoring systems are not
well-behaved (e.g. some bug in the formula causing scoring formula to
go backwards and many other things).

That is why not all Similarities are in RandomSImilarity, because some
still cannot meet that bar and have crazy behavior, some may be
unfixable! But the tests found tons of bugs in these formulas, where
bad things happen when their expected distribution is not met, or
where features of lucene like boost don't play well with them. It is
important to have tests, somewhere, somehow, that will find these
things.

Making all the tests wimpy so that BooleanSimilarity can be in there
doesn't make any sense to me.


On Sun, Nov 13, 2016 at 8:58 AM, Robert Muir  wrote:
> I think these fixes are not the best way to go.
>
> Overall, these commits destroy large features of lucene's test suite,
> for testing that similarities are well-behaved and disables them.
> Previously, they were randomized across all similarities: This is the
> whole point of doing that.
>
> But why add BooleanSimilarity to the random list at all, when any test
> failures it causes (because of its own brokenness!), will be disabled
> to use only BM25 This hurts all of lucene.
>
> Instead, I think all these test changes should be reverted and either:
> 1) BooleanSimilarity should be fixed to not be so brain-dead: e.g. why
> does it ignore index-time boost but not query-time boost? This is
> buggy are some of the reasons these tests fail.
>
> or
>
> 2) Remove BooleanSimilarity from the randomized testing setup here.
>
>
>
>
> On Sat, Nov 12, 2016 at 1:29 PM,   wrote:
>> Repository: lucene-solr
>> Updated Branches:
>>   refs/heads/master 4d9451034 -> 2bc1d2761
>>
>>
>> LUCENE-7555: use BM25Similarity for this test
>>
>>
>> Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
>> Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2bc1d276
>> Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2bc1d276
>> Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2bc1d276
>>
>> Branch: refs/heads/master
>> Commit: 2bc1d2761f593b2ceea19b3195e0cb430318ceaa
>> Parents: 4d94510
>> Author: Mike McCandless 
>> Authored: Sat Nov 12 13:28:57 2016 -0500
>> Committer: Mike McCandless 
>> Committed: Sat Nov 12 13:28:57 2016 -0500
>>
>> --
>>  .../src/test/org/apache/solr/uninverting/TestFieldCacheSort.java | 4 +++-
>>  1 file changed, 3 insertions(+), 1 deletion(-)
>> --
>>
>>
>> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2bc1d276/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>> --
>> diff --git 
>> a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java 
>> b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>> index d53f610..4755c8f 100644
>> --- a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>> +++ b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
>> @@ -50,10 +50,11 @@ import org.apache.lucene.search.Sort;
>>  import org.apache.lucene.search.SortField;
>>  import org.apache.lucene.search.TermQuery;
>>  import org.apache.lucene.search.TopDocs;
>> +import org.apache.lucene.search.similarities.BM25Similarity;
>>  import org.apache.lucene.store.Directory;
>> -import org.apache.solr.uninverting.UninvertingReader.Type;
>>  import org.apache.lucene.util.LuceneTestCase;
>>  import org.apache.lucene.util.TestUtil;
>> +import org.apache.solr.uninverting.UninvertingReader.Type;
>>
>>  /*
>>   * Tests sorting (but with fieldcache instead of docvalues)
>> @@ -434,6 +435,7 @@ public class TestFieldCacheSort extends LuceneTestCase {
>>  writer.close();
>>
>>  IndexSearcher searcher = newSearcher(ir);
>> +searcher.setSimilarity(new BM25Similarity());
>>  Sort sort = new Sort(new SortField(null, SortField.Type.SCORE, true));
>>
>>  TopDocs actual = searcher.search(new TermQuery(new Term("value", 
>> "foo")), 10, sort);
>>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: lucene-solr:master: LUCENE-7555: use BM25Similarity for this test

[Robert Muir]

I think these fixes are not the best way to go.

Overall, these commits destroy large features of lucene's test suite,
for testing that similarities are well-behaved and disables them.
Previously, they were randomized across all similarities: This is the
whole point of doing that.

But why add BooleanSimilarity to the random list at all, when any test
failures it causes (because of its own brokenness!), will be disabled
to use only BM25 This hurts all of lucene.

Instead, I think all these test changes should be reverted and either:
1) BooleanSimilarity should be fixed to not be so brain-dead: e.g. why
does it ignore index-time boost but not query-time boost? This is
buggy are some of the reasons these tests fail.

or

2) Remove BooleanSimilarity from the randomized testing setup here.




On Sat, Nov 12, 2016 at 1:29 PM,   wrote:
> Repository: lucene-solr
> Updated Branches:
>   refs/heads/master 4d9451034 -> 2bc1d2761
>
>
> LUCENE-7555: use BM25Similarity for this test
>
>
> Project: http://git-wip-us.apache.org/repos/asf/lucene-solr/repo
> Commit: http://git-wip-us.apache.org/repos/asf/lucene-solr/commit/2bc1d276
> Tree: http://git-wip-us.apache.org/repos/asf/lucene-solr/tree/2bc1d276
> Diff: http://git-wip-us.apache.org/repos/asf/lucene-solr/diff/2bc1d276
>
> Branch: refs/heads/master
> Commit: 2bc1d2761f593b2ceea19b3195e0cb430318ceaa
> Parents: 4d94510
> Author: Mike McCandless 
> Authored: Sat Nov 12 13:28:57 2016 -0500
> Committer: Mike McCandless 
> Committed: Sat Nov 12 13:28:57 2016 -0500
>
> --
>  .../src/test/org/apache/solr/uninverting/TestFieldCacheSort.java | 4 +++-
>  1 file changed, 3 insertions(+), 1 deletion(-)
> --
>
>
> http://git-wip-us.apache.org/repos/asf/lucene-solr/blob/2bc1d276/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
> --
> diff --git 
> a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java 
> b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
> index d53f610..4755c8f 100644
> --- a/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
> +++ b/solr/core/src/test/org/apache/solr/uninverting/TestFieldCacheSort.java
> @@ -50,10 +50,11 @@ import org.apache.lucene.search.Sort;
>  import org.apache.lucene.search.SortField;
>  import org.apache.lucene.search.TermQuery;
>  import org.apache.lucene.search.TopDocs;
> +import org.apache.lucene.search.similarities.BM25Similarity;
>  import org.apache.lucene.store.Directory;
> -import org.apache.solr.uninverting.UninvertingReader.Type;
>  import org.apache.lucene.util.LuceneTestCase;
>  import org.apache.lucene.util.TestUtil;
> +import org.apache.solr.uninverting.UninvertingReader.Type;
>
>  /*
>   * Tests sorting (but with fieldcache instead of docvalues)
> @@ -434,6 +435,7 @@ public class TestFieldCacheSort extends LuceneTestCase {
>  writer.close();
>
>  IndexSearcher searcher = newSearcher(ir);
> +searcher.setSimilarity(new BM25Similarity());
>  Sort sort = new Sort(new SortField(null, SortField.Type.SCORE, true));
>
>  TopDocs actual = searcher.search(new TermQuery(new Term("value", 
> "foo")), 10, sort);
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9752) Reliable test failure in TestFieldCacheSort (Solr copy)

[Steve Rowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Steve Rowe resolved SOLR-9752.
--
Resolution: Duplicate
  Assignee: Steve Rowe

This is a duplicate of LUCENE-7555; the seed reported here no longer reproduces 
for me after the fix committed there.

This test, like several others (see LUCENE-7554, LUCENE-7556, LUCENE-7557, 
LUCENE-7558), makes assumptions about scoring that depends on the (randomized) 
similarity - this issue was exposed by the introduction of BooleanSimilarity a 
few days ago (LUCENE-5867).  See [~mikemccand]'s [comment on 
LUCENE-7557|https://issues.apache.org/jira/browse/LUCENE-7557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15659833#comment-15659833]
 for more details.


> Reliable test failure in TestFieldCacheSort (Solr copy)
> ---
>
> Key: SOLR-9752
> URL: https://issues.apache.org/jira/browse/SOLR-9752
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
> Environment: OS X, Sierra
>Reporter: Erick Erickson
>Assignee: Steve Rowe
>
> I found this to reliably fail on 6x, see the whole dump below. At
> least the 4 times I tried it. I hit this running the full test suite
> for SOLR-9166, but then reproduced it on a fresh 6x checkout. 
> It did NOT repro on trunk the one time I tried it there.
> OS X, Sierra
> Here's the test that failed, I didn't try other variants.
> ant test  -Dtestcase=TestFieldCacheSort
> -Dtests.method=testFieldScoreReverse -Dtests.seed=8A982858396AE681
> -Dtests.slow=true -Dtests.locale=es-VE -Dtests.timezone=Asia/Baku
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
> [junit4:pickseed] Seed property 'tests.seed' already defined: 8A982858396AE681
>[junit4]  says ciao! Master seed: 8A982858396AE681
>[junit4] Executing 1 suite with 1 JVM.
>[junit4]
>[junit4] Started J0 PID(19306@Ericks-MacBook-Pro.local).
>[junit4] Suite: org.apache.solr.uninverting.TestFieldCacheSort
>[junit4]   2> NOTE: reproduce with: ant test
> -Dtestcase=TestFieldCacheSort -Dtests.method=testFieldScoreReverse
> -Dtests.seed=8A982858396AE681 -Dtests.slow=true -Dtests.locale=es-VE
> -Dtests.timezone=Asia/Baku -Dtests.asserts=true
> -Dtests.file.encoding=UTF-8
>[junit4] FAILURE 3.18s | TestFieldCacheSort.testFieldScoreReverse <<<
>[junit4]> Throwable #1: java.lang.AssertionError: expected:<0>
> but was:<1>
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([8A982858396AE681:F15FD36FC8A76C02]:0)
>[junit4]> at
> org.apache.solr.uninverting.TestFieldCacheSort.testFieldScoreReverse(TestFieldCacheSort.java:445)
>[junit4]> at java.lang.Thread.run(Thread.java:745)
>[junit4]   2> NOTE: test params are: codec=Asserting(Lucene62):
> {value=FSTOrd50}, docValues:{}, maxPointsInLeafNode=1095,
> maxMBSortInHeap=5.032597879580006,
> sim=RandomSimilarity(queryNorm=false,coord=crazy):
> {value=org.apache.lucene.search.similarities.BooleanSimilarity@5077995e},
> locale=es-VE, timezone=Asia/Baku
>[junit4]   2> NOTE: Mac OS X 10.12 x86_64/Oracle Corporation
> 1.8.0_45 (64-bit)/cpus=8,threads=1,free=243860576,total=257425408
>[junit4]   2> NOTE: All tests run in this JVM: [TestFieldCacheSort]
>[junit4] Completed [1/1 (1!)] in 6.14s, 1 test, 1 failure <<< FAILURES!
>[junit4] Tests with failures [seed: 8A982858396AE681]:
>[junit4]   -
> org.apache.solr.uninverting.TestFieldCacheSort.testFieldScoreReverse
>[junit4] JVM J0: 0.57 .. 8.10 = 7.53s
>[junit4] Execution time total: 8.10 sec.
>[junit4] Tests summary: 1 suite, 1 test, 1 failure



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7555) TestFieldCacheSort.testFieldScoreReverse() failure

[Steve Rowe (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661517#comment-15661517
 ] 

Steve Rowe commented on LUCENE-7555:


bq. Is this also SOLR-9752?

Yes, it is - I've commented there and closed it as a duplicate of this issue.

> TestFieldCacheSort.testFieldScoreReverse() failure
> --
>
> Key: LUCENE-7555
> URL: https://issues.apache.org/jira/browse/LUCENE-7555
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/other
>Reporter: Steve Rowe
>Assignee: Steve Rowe
> Fix For: master (7.0), 6.4
>
>
> My Jenkins found a reproducing seed on branch_6x:
> {noformat}
>   [junit4] Suite: org.apache.lucene.uninverting.TestFieldCacheSort
>   [junit4]   2> NOTE: reproduce with: ant test  -Dtestcase=TestFieldCacheSort 
> -Dtests.method=testFieldScoreReverse -Dtests.seed=DDD3900D2520B584 
> -Dtests.slow=true -Dtests.locale=ko -Dtests.timezone=America/Adak 
> -Dtests.asserts=true -Dtests.file.encoding=UTF-8
>   [junit4] FAILURE 0.07s J3 | TestFieldCacheSort.testFieldScoreReverse <<<
>   [junit4]> Throwable #1: java.lang.AssertionError: expected:<0> but 
> was:<1>
>   [junit4]>   at 
> __randomizedtesting.SeedInfo.seed([DDD3900D2520B584:A6146B3AD4ED3F07]:0)
>   [junit4]>   at 
> org.apache.lucene.uninverting.TestFieldCacheSort.testFieldScoreReverse(TestFieldCacheSort.java:445)
>   [junit4]>   at java.lang.Thread.run(Thread.java:745)
>   [junit4]   2> NOTE: test params are: codec=Asserting(Lucene62): 
> {t=PostingsFormat(name=Asserting), 
> string=PostingsFormat(name=LuceneFixedGap), f=BlockTreeOrds(blocksize=128), 
> id=PostingsFormat(name=Memory doPackFST= false), 
> body=BlockTreeOrds(blocksize=128), value=PostingsFormat(name=LuceneFixedGap), 
> tievalue=PostingsFormat(name=LuceneFixedGap)}, docValues:{}, 
> maxPointsInLeafNode=1889, maxMBSortInHeap=6.363051260515641, 
> sim=RandomSimilarity(queryNorm=false,coord=crazy): {contents=DFR 
> I(F)3(800.0), body=DFR I(F)2, 
> value=org.apache.lucene.search.similarities.BooleanSimilarity@2f181fb4}, 
> locale=ko, timezone=America/Adak
>   [junit4]   2> NOTE: Linux 4.1.0-custom2-amd64 amd64/Oracle Corporation 
> 1.8.0_77 (64-bit)/cpus=16,threads=1,free=488108520,total=514850816
>   [junit4]   2> NOTE: All tests run in this JVM: [TestMultiPassIndexSplitter, 
> TestLazyDocument, TestFieldCacheSort]
>   [junit4] Completed [18/23 (1!)] on J3 in 3.69s, 60 tests, 1 failure <<< 
> FAILURES!
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.x-Windows (32bit/jdk1.8.0_102) - Build # 558 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-6.x-Windows/558/
Java: 32bit/jdk1.8.0_102 -client -XX:+UseG1GC

1 tests failed.
FAILED:  
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure

Error Message:
The data directory was not cleaned up on unload after a failed core reload

Stack Trace:
java.lang.AssertionError: The data directory was not cleaned up on unload after 
a failed core reload
at 
__randomizedtesting.SeedInfo.seed([367F0480E4B4C310:4DB6A6CCC796]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.handler.admin.CoreAdminHandlerTest.testDeleteInstanceDirAfterCreateFailure(CoreAdminHandlerTest.java:334)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 11175 lines...]
   [junit4] Suite: org.apache.solr.handler.admin.CoreAdminHandlerTest
   [junit4]   2> Creating dataDir: 

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6222 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6222/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([1D214591923009F4:759E70BB42AA1B18]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:140)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:304)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1713)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:957)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:811)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:462)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:802)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:852)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:863)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Comment Edited] (LUCENE-6989) Implement MMapDirectory unmapping for coming Java 9 changes

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661100#comment-15661100
 ] 

Uwe Schindler edited comment on LUCENE-6989 at 11/13/16 8:36 AM:
-

The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

UPDATE: The Java 7 workaround is a change to the {{nonNull}} check to use 
{{Objects.equals(Object,Object)}} with a bound {{null}} reference as 1st arg 
and later in the code invert the if/then/else logic (swap the 2 last args in 
{{MethodHandles.guardWithTest}}):

{code:java}
final MethodHandle nullTest = lookup.findStatic(Objects.class, "equals", 
methodType(boolean.class, Object.class, Object.class))
  .bindTo(null)
  .asType(methodType(boolean.class, cleanerClass));
{code}

I did not test / compile this code, was just hacked into this issue :-)


was (Author: thetaphi):
The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

UPDATE: The Java 7 workaround is a change to the nonNull check to use 
Objects.equals with a bound {{null}} reference as 1st arg:

{code:java}
final MethodHandle nonNullTest = lookup.findStatic(Objects.class, "equals", 
methodType(boolean.class, Object.class, Object.class))
  .bindTo(null)
  .asType(methodType(boolean.class, cleanerClass));
{code}

I did not test / compile this code, was just hacked into this issue :-)

> Implement MMapDirectory unmapping for coming Java 9 changes
> ---
>
> Key: LUCENE-6989
> URL: https://issues.apache.org/jira/browse/LUCENE-6989
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>  Labels: Java9
> Fix For: 6.0
>
> Attachments: LUCENE-6989-disable5x.patch, 
> LUCENE-6989-disable5x.patch, LUCENE-6989-v2.patch, LUCENE-6989.patch, 
> LUCENE-6989.patch, LUCENE-6989.patch, LUCENE-6989.patch
>
>
> Originally, the sun.misc.Cleaner interface was declared as "critical API" in 
> [JEP 260|http://openjdk.java.net/jeps/260 ]
> Unfortunately the decission was changed in favor of a oficially supported 
> {{java.lang.ref.Cleaner}} API. A side effect of this change is to move all 
> existing {{sun.misc.Cleaner}} APIs into a non-exported package. This causes 
> our forceful unmapping to no longer work, because we can get the cleaner 
> instance via reflection, but trying to invoke it will throw one of the new 
> Jigsaw RuntimeException because it is completely inaccessible. This will make 
> our forceful unmapping fail. There are also no changes in Garbage collector, 
> the problem still exists.
> For more information see this [mailing list 
> thread|http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/thread.html#38243].
> This commit will likely be done, making our unmapping efforts no longer 
> working. Alan Bateman is aware of this issue and will open a new issue at 
> OpenJDK to allow forceful unmapping without using the now private 
> sun.misc.Cleaner. The idea is to let the internal class sun.misc.Cleaner 
> implement the Runable interface, so we can simply cast to runable and call 
> the run() method to unmap. The code would then work. This will lead to minor 
> changes in our unmapper in MMapDirectory: An instanceof check and casting if 
> possible.
> I opened this issue to keep track and implement the changes as soon as 
> possible, so people will have working unmapping when java 9 comes out. 
> Current Lucene versions will no longer work with Java 9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-6989) Implement MMapDirectory unmapping for coming Java 9 changes

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661100#comment-15661100
 ] 

Uwe Schindler edited comment on LUCENE-6989 at 11/13/16 8:34 AM:
-

The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

UPDATE: The Java 7 workaround is a change to the nonNull check to use 
Objects.equals with a bound {{null}} reference as 1st arg:

{code:java}
final MethodHandle nonNullTest = lookup.findStatic(Objects.class, "equals", 
methodType(boolean.class, Object.class, Object.class))
  .bindTo(null)
  .asType(methodType(boolean.class, cleanerClass));
{code}

I did not test / compile this code, was just hacked into this issue :-)


was (Author: thetaphi):
The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

UPDATE: The Java 7 workaround is a change to the nonNull check to use 
Objects.equals with bound {{code}} as 1st arg:

{code:java}
final MethodHandle nonNullTest = lookup.findStatic(Objects.class, "equals", 
methodType(boolean.class, Object.class, Object.class))
  .bindTo(null)
  .asType(methodType(boolean.class, cleanerClass));
{code}

I did not test / compile this code, was just hacked into this issue :-)

> Implement MMapDirectory unmapping for coming Java 9 changes
> ---
>
> Key: LUCENE-6989
> URL: https://issues.apache.org/jira/browse/LUCENE-6989
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>  Labels: Java9
> Fix For: 6.0
>
> Attachments: LUCENE-6989-disable5x.patch, 
> LUCENE-6989-disable5x.patch, LUCENE-6989-v2.patch, LUCENE-6989.patch, 
> LUCENE-6989.patch, LUCENE-6989.patch, LUCENE-6989.patch
>
>
> Originally, the sun.misc.Cleaner interface was declared as "critical API" in 
> [JEP 260|http://openjdk.java.net/jeps/260 ]
> Unfortunately the decission was changed in favor of a oficially supported 
> {{java.lang.ref.Cleaner}} API. A side effect of this change is to move all 
> existing {{sun.misc.Cleaner}} APIs into a non-exported package. This causes 
> our forceful unmapping to no longer work, because we can get the cleaner 
> instance via reflection, but trying to invoke it will throw one of the new 
> Jigsaw RuntimeException because it is completely inaccessible. This will make 
> our forceful unmapping fail. There are also no changes in Garbage collector, 
> the problem still exists.
> For more information see this [mailing list 
> thread|http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/thread.html#38243].
> This commit will likely be done, making our unmapping efforts no longer 
> working. Alan Bateman is aware of this issue and will open a new issue at 
> OpenJDK to allow forceful unmapping without using the now private 
> sun.misc.Cleaner. The idea is to let the internal class sun.misc.Cleaner 
> implement the Runable interface, so we can simply cast to runable and call 
> the run() method to unmap. The code would then work. This will lead to minor 
> changes in our unmapper in MMapDirectory: An instanceof check and casting if 
> possible.
> I opened this issue to keep track and implement the changes as soon as 
> possible, so people will have working unmapping when java 9 comes out. 
> Current Lucene versions will no longer work with Java 9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-6989) Implement MMapDirectory unmapping for coming Java 9 changes

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661100#comment-15661100
 ] 

Uwe Schindler edited comment on LUCENE-6989 at 11/13/16 8:33 AM:
-

The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

UPDATE: The Java 7 workaround is a change to the nonNull check to use 
Objects.equals with bound {{code}} as 1st arg:

{code:java}
final MethodHandle nonNullTest = lookup.findStatic(Objects.class, "equals", 
methodType(boolean.class, Object.class, Object.class))
  .bindTo(null)
  .asType(methodType(boolean.class, cleanerClass));
{code}

I did not test / compile this code, was just hacked into this issue :-)


was (Author: thetaphi):
The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

> Implement MMapDirectory unmapping for coming Java 9 changes
> ---
>
> Key: LUCENE-6989
> URL: https://issues.apache.org/jira/browse/LUCENE-6989
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>  Labels: Java9
> Fix For: 6.0
>
> Attachments: LUCENE-6989-disable5x.patch, 
> LUCENE-6989-disable5x.patch, LUCENE-6989-v2.patch, LUCENE-6989.patch, 
> LUCENE-6989.patch, LUCENE-6989.patch, LUCENE-6989.patch
>
>
> Originally, the sun.misc.Cleaner interface was declared as "critical API" in 
> [JEP 260|http://openjdk.java.net/jeps/260 ]
> Unfortunately the decission was changed in favor of a oficially supported 
> {{java.lang.ref.Cleaner}} API. A side effect of this change is to move all 
> existing {{sun.misc.Cleaner}} APIs into a non-exported package. This causes 
> our forceful unmapping to no longer work, because we can get the cleaner 
> instance via reflection, but trying to invoke it will throw one of the new 
> Jigsaw RuntimeException because it is completely inaccessible. This will make 
> our forceful unmapping fail. There are also no changes in Garbage collector, 
> the problem still exists.
> For more information see this [mailing list 
> thread|http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/thread.html#38243].
> This commit will likely be done, making our unmapping efforts no longer 
> working. Alan Bateman is aware of this issue and will open a new issue at 
> OpenJDK to allow forceful unmapping without using the now private 
> sun.misc.Cleaner. The idea is to let the internal class sun.misc.Cleaner 
> implement the Runable interface, so we can simply cast to runable and call 
> the run() method to unmap. The code would then work. This will lead to minor 
> changes in our unmapper in MMapDirectory: An instanceof check and casting if 
> possible.
> I opened this issue to keep track and implement the changes as soon as 
> possible, so people will have working unmapping when java 9 comes out. 
> Current Lucene versions will no longer work with Java 9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6989) Implement MMapDirectory unmapping for coming Java 9 changes

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661100#comment-15661100
 ] 

Uwe Schindler commented on LUCENE-6989:
---

The problem with backporting is usage of Java 8 APIs: To allow backporting to 
5.5, we must change the unmapper bytecode, because {{Objects.nonNull(Object)}} 
does not exist in Java 7. We have to work around that with a hack (I had some 
in mind), I have to find the records.

> Implement MMapDirectory unmapping for coming Java 9 changes
> ---
>
> Key: LUCENE-6989
> URL: https://issues.apache.org/jira/browse/LUCENE-6989
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>  Labels: Java9
> Fix For: 6.0
>
> Attachments: LUCENE-6989-disable5x.patch, 
> LUCENE-6989-disable5x.patch, LUCENE-6989-v2.patch, LUCENE-6989.patch, 
> LUCENE-6989.patch, LUCENE-6989.patch, LUCENE-6989.patch
>
>
> Originally, the sun.misc.Cleaner interface was declared as "critical API" in 
> [JEP 260|http://openjdk.java.net/jeps/260 ]
> Unfortunately the decission was changed in favor of a oficially supported 
> {{java.lang.ref.Cleaner}} API. A side effect of this change is to move all 
> existing {{sun.misc.Cleaner}} APIs into a non-exported package. This causes 
> our forceful unmapping to no longer work, because we can get the cleaner 
> instance via reflection, but trying to invoke it will throw one of the new 
> Jigsaw RuntimeException because it is completely inaccessible. This will make 
> our forceful unmapping fail. There are also no changes in Garbage collector, 
> the problem still exists.
> For more information see this [mailing list 
> thread|http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/thread.html#38243].
> This commit will likely be done, making our unmapping efforts no longer 
> working. Alan Bateman is aware of this issue and will open a new issue at 
> OpenJDK to allow forceful unmapping without using the now private 
> sun.misc.Cleaner. The idea is to let the internal class sun.misc.Cleaner 
> implement the Runable interface, so we can simply cast to runable and call 
> the run() method to unmap. The code would then work. This will lead to minor 
> changes in our unmapper in MMapDirectory: An instanceof check and casting if 
> possible.
> I opened this issue to keep track and implement the changes as soon as 
> possible, so people will have working unmapping when java 9 comes out. 
> Current Lucene versions will no longer work with Java 9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6989) Implement MMapDirectory unmapping for coming Java 9 changes

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15661092#comment-15661092
 ] 

Uwe Schindler commented on LUCENE-6989:
---

This is all working, but I want to wait until Java 9 is released, because this 
hack could still break until the release of Java 9 (Jenkins will catch this).

Backporting to 5.5 should be possible, but I won't do this. Lucene 5.x is not 
compatible to Java 9 for other reasons, too, because it has no Jigsaw support. 
So this would not really help.

> Implement MMapDirectory unmapping for coming Java 9 changes
> ---
>
> Key: LUCENE-6989
> URL: https://issues.apache.org/jira/browse/LUCENE-6989
> Project: Lucene - Core
>  Issue Type: Task
>  Components: core/store
>Reporter: Uwe Schindler
>Assignee: Uwe Schindler
>  Labels: Java9
> Fix For: 6.0
>
> Attachments: LUCENE-6989-disable5x.patch, 
> LUCENE-6989-disable5x.patch, LUCENE-6989-v2.patch, LUCENE-6989.patch, 
> LUCENE-6989.patch, LUCENE-6989.patch, LUCENE-6989.patch
>
>
> Originally, the sun.misc.Cleaner interface was declared as "critical API" in 
> [JEP 260|http://openjdk.java.net/jeps/260 ]
> Unfortunately the decission was changed in favor of a oficially supported 
> {{java.lang.ref.Cleaner}} API. A side effect of this change is to move all 
> existing {{sun.misc.Cleaner}} APIs into a non-exported package. This causes 
> our forceful unmapping to no longer work, because we can get the cleaner 
> instance via reflection, but trying to invoke it will throw one of the new 
> Jigsaw RuntimeException because it is completely inaccessible. This will make 
> our forceful unmapping fail. There are also no changes in Garbage collector, 
> the problem still exists.
> For more information see this [mailing list 
> thread|http://mail.openjdk.java.net/pipermail/core-libs-dev/2016-January/thread.html#38243].
> This commit will likely be done, making our unmapping efforts no longer 
> working. Alan Bateman is aware of this issue and will open a new issue at 
> OpenJDK to allow forceful unmapping without using the now private 
> sun.misc.Cleaner. The idea is to let the internal class sun.misc.Cleaner 
> implement the Runable interface, so we can simply cast to runable and call 
> the run() method to unmap. The code would then work. This will lead to minor 
> changes in our unmapper in MMapDirectory: An instanceof check and casting if 
> possible.
> I opened this issue to keep track and implement the changes as soon as 
> possible, so people will have working unmapping when java 9 comes out. 
> Current Lucene versions will no longer work with Java 9.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org