[jira] [Commented] (LUCENE-6330) BooleanScorer should not call score() when scores are not needed
[ https://issues.apache.org/jira/browse/LUCENE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519148#comment-14519148 ] Robert Muir commented on LUCENE-6330: - +1 BooleanScorer should not call score() when scores are not needed Key: LUCENE-6330 URL: https://issues.apache.org/jira/browse/LUCENE-6330 Project: Lucene - Core Issue Type: Task Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6330.patch BooleanScorer (and maybe others?) still call score() when scores are not needed. While this does not make results wrong, it is sub-optimal since it still forces to decode norms while they are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[
https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adrien Grand updated LUCENE-6458:
-
Attachment: LUCENE-6458.patch
Here is a patch, it is quite similar to the old auto rewrite except that it
rewrites per segment and only consumes the filtered terms enum once. Queries
are executed as regular disjunctions when there are 50 matching terms or less.
{noformat}
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
Prefix3 113.17 (1.7%) 88.55 (2.7%)
-21.8% ( -25% - -17%)
Wildcard 37.43 (2.0%) 36.26 (3.2%)
-3.1% ( -8% -2%)
HighSpanNear4.30 (2.6%)4.24 (4.0%)
-1.6% ( -7% -5%)
OrHighNotLow 71.52 (1.5%) 70.51 (3.1%)
-1.4% ( -5% -3%)
HighSloppyPhrase 20.60 (6.3%) 20.34 (7.6%)
-1.3% ( -14% - 13%)
OrHighNotMed 96.14 (2.0%) 95.11 (2.8%)
-1.1% ( -5% -3%)
MedPhrase 23.49 (1.8%) 23.30 (3.5%)
-0.8% ( -6% -4%)
Respell 62.25 (8.9%) 62.01 (7.4%)
-0.4% ( -15% - 17%)
AndHighHigh 52.43 (0.7%) 52.27 (1.1%)
-0.3% ( -2% -1%)
OrNotHighHigh 26.08 (3.5%) 26.02 (1.0%)
-0.2% ( -4% -4%)
OrHighNotHigh 61.96 (2.0%) 61.85 (2.1%)
-0.2% ( -4% -4%)
IntNRQ8.03 (3.1%)8.02 (2.6%)
-0.2% ( -5% -5%)
LowTerm 783.62 (4.9%) 783.25 (4.5%)
-0.0% ( -9% -9%)
MedSpanNear 18.77 (1.9%) 18.76 (3.6%)
-0.0% ( -5% -5%)
LowSpanNear 14.49 (2.5%) 14.49 (2.6%)
-0.0% ( -4% -5%)
MedTerm 237.81 (2.1%) 237.76 (3.0%)
-0.0% ( -4% -5%)
PKLookup 266.15 (2.5%) 266.38 (2.5%)
0.1% ( -4% -5%)
OrHighMed 50.61 (6.0%) 50.68 (6.1%)
0.1% ( -11% - 13%)
Fuzzy2 19.87 (4.4%) 19.92 (7.8%)
0.2% ( -11% - 12%)
OrNotHighMed 90.03 (1.1%) 90.25 (0.8%)
0.2% ( -1% -2%)
HighPhrase 15.56 (2.0%) 15.61 (2.7%)
0.3% ( -4% -5%)
MedSloppyPhrase 252.97 (5.2%) 253.93 (4.3%)
0.4% ( -8% - 10%)
LowPhrase8.16 (1.7%)8.21 (1.9%)
0.6% ( -2% -4%)
HighTerm 115.17 (2.4%) 116.05 (2.7%)
0.8% ( -4% -6%)
OrHighHigh 25.19 (5.7%) 25.45 (6.4%)
1.0% ( -10% - 13%)
OrHighLow 42.12 (7.5%) 42.60 (6.9%)
1.1% ( -12% - 16%)
LowSloppyPhrase 129.20 (1.6%) 130.68 (2.0%)
1.2% ( -2% -4%)
AndHighMed 231.64 (1.3%) 235.28 (2.1%)
1.6% ( -1% -4%)
AndHighLow 733.51 (3.9%) 751.08 (3.5%)
2.4% ( -4% - 10%)
Fuzzy1 85.42 (17.0%) 91.04 (5.9%)
6.6% ( -13% - 35%)
OrNotHighLow 893.55 (2.9%) 962.35 (4.6%)
7.7% ( 0% - 15%)
{noformat}
I was hoping it would kick in for numeric range queries but unfortunately they
often need to match hundreds of terms. I'm wondering if it would be different
for auto-prefix.
Prefix3 and Wildcard are a bit slower because these ones get actually executed
as regular disjunctions. I think the slowdown is fair given that it also
requires less memory and provides true skipping support (which the benchmark
doesn't use).
MultiTermQuery's FILTER rewrite method should support skipping whenever
possible
Key: LUCENE-6458
URL: https://issues.apache.org/jira/browse/LUCENE-6458
Project: Lucene - Core
Issue Type: Improvement
Reporter: Adrien Grand
Assignee: Adrien Grand
Priority: Minor
Attachments: LUCENE-6458.patch
Today MultiTermQuery's FILTER rewrite always builds a bit set fom all
matching terms. This means that we need to consume the entire postings lists
of all matching terms. Instead we should try to execute like regular
disjunctions when there are few terms.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional
[jira] [Comment Edited] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[
https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519160#comment-14519160
]
Adrien Grand edited comment on LUCENE-6458 at 4/29/15 11:34 AM:
Here is a patch, it is quite similar to the old auto rewrite except that it
rewrites per segment and only consumes the filtered terms enum once. Queries
are executed as regular disjunctions when there are 50 matching terms or less.
{noformat}
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
Wildcard 76.93 (1.7%) 66.55 (4.1%)
-13.5% ( -18% - -7%)
Prefix3 99.69 (1.8%) 88.80 (2.3%)
-10.9% ( -14% - -6%)
OrHighMed 76.77 (3.7%) 76.26 (3.7%)
-0.7% ( -7% -6%)
OrHighNotHigh 37.88 (1.7%) 37.73 (2.2%)
-0.4% ( -4% -3%)
MedTerm 306.74 (1.4%) 305.54 (1.5%)
-0.4% ( -3% -2%)
OrHighHigh 36.17 (4.5%) 36.05 (4.0%)
-0.3% ( -8% -8%)
HighTerm 120.67 (1.2%) 120.37 (1.7%)
-0.2% ( -3% -2%)
MedSloppyPhrase 36.30 (2.9%) 36.25 (2.8%)
-0.1% ( -5% -5%)
IntNRQ8.64 (2.9%)8.63 (2.6%)
-0.1% ( -5% -5%)
LowSpanNear 70.11 (1.8%) 70.13 (2.2%)
0.0% ( -3% -4%)
HighSpanNear 17.55 (2.8%) 17.56 (3.0%)
0.1% ( -5% -6%)
OrHighNotMed 81.45 (1.8%) 81.51 (2.2%)
0.1% ( -3% -4%)
LowPhrase 14.47 (2.7%) 14.50 (3.0%)
0.2% ( -5% -6%)
MedSpanNear 120.55 (2.2%) 120.86 (2.0%)
0.3% ( -3% -4%)
AndHighHigh 58.08 (2.5%) 58.24 (2.6%)
0.3% ( -4% -5%)
LowSloppyPhrase 62.42 (4.3%) 62.60 (4.4%)
0.3% ( -8% -9%)
OrHighNotLow 76.06 (1.9%) 76.36 (2.4%)
0.4% ( -3% -4%)
Respell 72.86 (3.9%) 73.17 (2.9%)
0.4% ( -6% -7%)
OrNotHighHigh 50.07 (1.5%) 50.30 (1.2%)
0.5% ( -2% -3%)
HighSloppyPhrase 24.92 (6.4%) 25.05 (6.5%)
0.5% ( -11% - 14%)
OrHighLow 68.75 (4.6%) 69.17 (4.1%)
0.6% ( -7% -9%)
HighPhrase 20.89 (2.5%) 21.04 (1.8%)
0.7% ( -3% -5%)
OrNotHighMed 179.02 (1.9%) 180.37 (1.4%)
0.8% ( -2% -4%)
PKLookup 263.21 (2.8%) 265.42 (3.0%)
0.8% ( -4% -6%)
MedPhrase 34.60 (3.6%) 34.94 (3.4%)
1.0% ( -5% -8%)
LowTerm 780.71 (3.2%) 790.04 (4.2%)
1.2% ( -5% -8%)
OrNotHighLow 1459.46 (3.5%) 1480.76 (5.0%)
1.5% ( -6% - 10%)
AndHighMed 255.15 (2.6%) 258.93 (2.4%)
1.5% ( -3% -6%)
Fuzzy1 77.69 (8.9%) 79.12 (7.7%)
1.8% ( -13% - 20%)
AndHighLow 961.32 (3.9%) 980.23 (3.5%)
2.0% ( -5% -9%)
Fuzzy2 24.48 (7.9%) 25.19 (7.4%)
2.9% ( -11% - 19%)
{noformat}
I was hoping it would kick in for numeric range queries but unfortunately they
often need to match hundreds of terms. I'm wondering if it would be different
for auto-prefix.
Prefix3 and Wildcard are a bit slower because these ones get actually executed
as regular disjunctions. I think the slowdown is fair given that it also
requires less memory and provides true skipping support (which the benchmark
doesn't use).
was (Author: jpountz):
Here is a patch, it is quite similar to the old auto rewrite except that it
rewrites per segment and only consumes the filtered terms enum once. Queries
are executed as regular disjunctions when there are 50 matching terms or less.
{noformat}
TaskQPS baseline StdDev QPS patch StdDev
Pct diff
Prefix3 113.17 (1.7%) 88.55 (2.7%)
-21.8% ( -25% - -17%)
Wildcard 37.43 (2.0%) 36.26 (3.2%)
-3.1% ( -8% -2%)
HighSpanNear4.30 (2.6%)4.24 (4.0%)
-1.6% ( -7% -5%)
OrHighNotLow 71.52 (1.5%) 70.51 (3.1%)
-1.4% ( -5% -3%)
HighSloppyPhrase 20.60 (6.3%) 20.34 (7.6%)
-1.3% ( -14% - 13%)
[jira] [Commented] (LUCENE-6083) Span containing/contained queries
[ https://issues.apache.org/jira/browse/LUCENE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519173#comment-14519173 ] ASF subversion and git services commented on LUCENE-6083: - Commit 1676716 from [~rcmuir] in branch 'dev/trunk' [ https://svn.apache.org/r1676716 ] LUCENE-6083: SpanWithinQuery and SpanContainingQuery Span containing/contained queries - Key: LUCENE-6083 URL: https://issues.apache.org/jira/browse/LUCENE-6083 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Attachments: LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch SpanContainingQuery reducing a spans to where it is containing another spans. SpanContainedQuery reducing a spans to where it is contained in another spans. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_45) - Build # 12499 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12499/
Java: 64bit/jdk1.8.0_45 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC
3 tests failed.
FAILED:
junit.framework.TestSuite.org.apache.solr.cloud.TestMiniSolrCloudCluster
Error Message:
Suite timeout exceeded (= 720 msec).
Stack Trace:
java.lang.Exception: Suite timeout exceeded (= 720 msec).
at __randomizedtesting.SeedInfo.seed([E4D72758D4844D12]:0)
FAILED: junit.framework.TestSuite.org.apache.solr.core.TestQuerySenderListener
Error Message:
Some resources were not closed, shutdown, or released.
Stack Trace:
java.lang.AssertionError: Some resources were not closed, shutdown, or released.
at __randomizedtesting.SeedInfo.seed([E4D72758D4844D12]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
at sun.reflect.GeneratedMethodAccessor40.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:799)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at java.lang.Thread.run(Thread.java:745)
FAILED: org.apache.solr.cloud.TestMiniSolrCloudCluster.testErrorsInShutdown
Error Message:
Test abandoned because suite timeout was reached.
Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([E4D72758D4844D12]:0)
Build Log:
[...truncated 10802 lines...]
[junit4] Suite: org.apache.solr.cloud.TestMiniSolrCloudCluster
[junit4] 2 393923 T2851 oasc.ZkTestServer.run STARTING ZK TEST SERVER
[junit4] 2 393923 T2852 oasc.ZkTestServer$2$1.setClientPort client
port:0.0.0.0/0.0.0.0:0
[junit4] 2 393923 T2852 oasc.ZkTestServer$ZKServerMain.runFromConfig
Starting server
[junit4] 2 394023 T2851 oasc.ZkTestServer.run start zk server on
port:59880
[junit4] 2 394086 T2861 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2 394087 T2862 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2 394088 T2860 oejs.Server.doStart jetty-9.2.10.v20150310
[junit4] 2 394091 T2860 oejsh.ContextHandler.doStart Started
o.e.j.s.ServletContextHandler@378e65b2{/solr,null,AVAILABLE}
[junit4] 2 394096 T2860 oejs.AbstractConnector.doStart Started
ServerConnector@6a2cd6ad{HTTP/1.1}{127.0.0.1:45239}
[junit4] 2 394097 T2860 oejs.Server.doStart Started @395001ms
[junit4] 2 394097 T2860 oascse.JettySolrRunner$1.lifeCycleStarted Jetty
properties: {hostContext=/solr, hostPort=45239}
[junit4] 2 394098 T2860 oass.SolrDispatchFilter.init
SolrDispatchFilter.init()sun.misc.Launcher$AppClassLoader@4e0e2f2a
[junit4] 2 394098 T2860 oasc.SolrResourceLoader.init new
SolrResourceLoader for directory:
'/home/jenkins/workspace/Lucene-Solr-trunk-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.TestMiniSolrCloudCluster
E4D72758D4844D12-001/tempDir-001/'
[jira] [Created] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
Adrien Grand created LUCENE-6458: Summary: MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[ https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6458: - Attachment: (was: LUCENE-6458.patch) MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6330) BooleanScorer should not call score() when scores are not needed
[ https://issues.apache.org/jira/browse/LUCENE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519177#comment-14519177 ] ASF subversion and git services commented on LUCENE-6330: - Commit 1676718 from [~jpountz] in branch 'dev/trunk' [ https://svn.apache.org/r1676718 ] LUCENE-6330: BS1 should not decode norms when scores are not needed. BooleanScorer should not call score() when scores are not needed Key: LUCENE-6330 URL: https://issues.apache.org/jira/browse/LUCENE-6330 Project: Lucene - Core Issue Type: Task Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6330.patch BooleanScorer (and maybe others?) still call score() when scores are not needed. While this does not make results wrong, it is sub-optimal since it still forces to decode norms while they are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7485) replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java
Christine Poerschke created SOLR-7485: - Summary: replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java Key: SOLR-7485 URL: https://issues.apache.org/jira/browse/SOLR-7485 Project: Solr Issue Type: Improvement Reporter: Christine Poerschke Priority: Minor various other tests already use ShardParams.SHARDS_INFO e.g. TestDistributedSearch.java -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7485) replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java
[ https://issues.apache.org/jira/browse/SOLR-7485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519132#comment-14519132 ] ASF GitHub Bot commented on SOLR-7485: -- GitHub user cpoerschke opened a pull request: https://github.com/apache/lucene-solr/pull/145 SOLR-7485: replace shards.info with ShardParams.SHARDS_INFO for https://issues.apache.org/jira/i#browse/SOLR-7485 You can merge this pull request into a Git repository by running: $ git pull https://github.com/bloomberg/lucene-solr trunk-shards.info Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/145.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #145 commit c53216fe38a56246ebc47e3e89134a217f60d924 Author: Christine Poerschke cpoersc...@bloomberg.net Date: 2015-01-15T17:11:50Z SOLR-: replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java various other tests already use ShardParams.SHARDS_INFO e.g. TestDistributedSearch.java replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java -- Key: SOLR-7485 URL: https://issues.apache.org/jira/browse/SOLR-7485 Project: Solr Issue Type: Improvement Reporter: Christine Poerschke Priority: Minor various other tests already use ShardParams.SHARDS_INFO e.g. TestDistributedSearch.java -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request: SOLR-7485: replace shards.info with Sh...
GitHub user cpoerschke opened a pull request: https://github.com/apache/lucene-solr/pull/145 SOLR-7485: replace shards.info with ShardParams.SHARDS_INFO for https://issues.apache.org/jira/i#browse/SOLR-7485 You can merge this pull request into a Git repository by running: $ git pull https://github.com/bloomberg/lucene-solr trunk-shards.info Alternatively you can review and apply these changes as the patch at: https://github.com/apache/lucene-solr/pull/145.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #145 commit c53216fe38a56246ebc47e3e89134a217f60d924 Author: Christine Poerschke cpoersc...@bloomberg.net Date: 2015-01-15T17:11:50Z SOLR-: replace shards.info with ShardParams.SHARDS_INFO in TestTolerantSearch.java and CloudSolrClientTest.java various other tests already use ShardParams.SHARDS_INFO e.g. TestDistributedSearch.java --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[ https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6458: - Attachment: LUCENE-6458.patch MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6458.patch Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6330) BooleanScorer should not call score() when scores are not needed
[ https://issues.apache.org/jira/browse/LUCENE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519184#comment-14519184 ] ASF subversion and git services commented on LUCENE-6330: - Commit 1676720 from [~jpountz] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676720 ] LUCENE-6330: BS1 should not decode norms when scores are not needed. BooleanScorer should not call score() when scores are not needed Key: LUCENE-6330 URL: https://issues.apache.org/jira/browse/LUCENE-6330 Project: Lucene - Core Issue Type: Task Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6330.patch BooleanScorer (and maybe others?) still call score() when scores are not needed. While this does not make results wrong, it is sub-optimal since it still forces to decode norms while they are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-6330) BooleanScorer should not call score() when scores are not needed
[ https://issues.apache.org/jira/browse/LUCENE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-6330. -- Resolution: Fixed BooleanScorer should not call score() when scores are not needed Key: LUCENE-6330 URL: https://issues.apache.org/jira/browse/LUCENE-6330 Project: Lucene - Core Issue Type: Task Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6330.patch BooleanScorer (and maybe others?) still call score() when scores are not needed. While this does not make results wrong, it is sub-optimal since it still forces to decode norms while they are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7349) Cleanup or fix cloud-dev scripts
[ https://issues.apache.org/jira/browse/SOLR-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519332#comment-14519332 ] ASF subversion and git services commented on SOLR-7349: --- Commit 1676740 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1676740 ] SOLR-7349: svn ignore extra server folders Cleanup or fix cloud-dev scripts Key: SOLR-7349 URL: https://issues.apache.org/jira/browse/SOLR-7349 Project: Solr Issue Type: Improvement Components: scripts and tools Reporter: Ramkumar Aiyengar Assignee: Mark Miller Priority: Minor Fix For: 5.2 Attachments: SOLR-7349.patch, SOLR-7349.patch With Jetty 9, cloud-dev scripts no longer work in trunk, we need to either clean up or fix them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[ https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519316#comment-14519316 ] David Smiley commented on LUCENE-6458: -- Adrien, how did you arrive at BOOLEAN_REWRITE_THRESHOLD=50 ? This reminds me of when I was working on the Solr Terms QParser that supports 3-4 different options, to include BooleanQuery TermsQuery. I wanted to have it automatically use a BooleanQuery at a low term threshold but I wasn't sure what to use so I didn't bother, and I didn't have time to do benchmarks then. In hind-site, any hunch value (64?) would have been better then always choosing TermsQuery no matter what. I have a feeling that the appropriate threshold is a function of the number of indexed terms, instead of just a constant. MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6458.patch Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-6083) Span containing/contained queries
[ https://issues.apache.org/jira/browse/LUCENE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Muir resolved LUCENE-6083. - Resolution: Fixed Fix Version/s: 5.2 Trunk Span containing/contained queries - Key: LUCENE-6083 URL: https://issues.apache.org/jira/browse/LUCENE-6083 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch SpanContainingQuery reducing a spans to where it is containing another spans. SpanContainedQuery reducing a spans to where it is contained in another spans. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6083) Span containing/contained queries
[ https://issues.apache.org/jira/browse/LUCENE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519190#comment-14519190 ] ASF subversion and git services commented on LUCENE-6083: - Commit 1676723 from [~rcmuir] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676723 ] LUCENE-6083: SpanWithinQuery and SpanContainingQuery Span containing/contained queries - Key: LUCENE-6083 URL: https://issues.apache.org/jira/browse/LUCENE-6083 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch SpanContainingQuery reducing a spans to where it is containing another spans. SpanContainedQuery reducing a spans to where it is contained in another spans. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yonik Seeley updated SOLR-7477: --- Attachment: SOLR-7477.patch Here's a patch that implements excludeTags It was a little trickier given that someone can excludeTags at any point in the facet hierarchy. To support this I basically had to start keeping track of of filters at each facet level so the current facet domain could be reconstructed (minus the excluded filters). multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6083) Span containing/contained queries
[ https://issues.apache.org/jira/browse/LUCENE-6083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519217#comment-14519217 ] Robert Muir commented on LUCENE-6083: - Paul, thank you for the work here! Span containing/contained queries - Key: LUCENE-6083 URL: https://issues.apache.org/jira/browse/LUCENE-6083 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch, LUCENE-6083.patch SpanContainingQuery reducing a spans to where it is containing another spans. SpanContainedQuery reducing a spans to where it is contained in another spans. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6427) BitSet fixes - assert on presence of 'ghost bits' and others
[ https://issues.apache.org/jira/browse/LUCENE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518853#comment-14518853 ] ASF GitHub Bot commented on LUCENE-6427: Github user LucVL closed the pull request at: https://github.com/apache/lucene-solr/pull/142 BitSet fixes - assert on presence of 'ghost bits' and others Key: LUCENE-6427 URL: https://issues.apache.org/jira/browse/LUCENE-6427 Project: Lucene - Core Issue Type: Bug Components: core/other Reporter: Luc Vanlerberghe Fix For: Trunk, 5.2 Fixes after reviewing org.apache.lucene.util.FixedBitSet, LongBitSet and corresponding tests: * Some methods rely on the fact that no bits are set after numBits (what I call 'ghost' bits here). ** cardinality, nextSetBit, intersects and others may yield wrong results ** If ghost bits are present, they may become visible after ensureCapacity is called. ** The tests deliberately create bitsets with ghost bits, but then do not detect these failures * FixedBitSet.cardinality scans the complete backing array, even if only numWords are in use -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6330) BooleanScorer should not call score() when scores are not needed
[ https://issues.apache.org/jira/browse/LUCENE-6330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-6330: - Attachment: LUCENE-6330.patch Here is a patch, it makes BS1 wrap sub scorers similarly to ConstantScoreQuery's BulkScorer when scores are not needed so that calling score on sub scorers (which it does for every document) does not need to eg. decode norms. BooleanScorer should not call score() when scores are not needed Key: LUCENE-6330 URL: https://issues.apache.org/jira/browse/LUCENE-6330 Project: Lucene - Core Issue Type: Task Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Fix For: Trunk, 5.2 Attachments: LUCENE-6330.patch BooleanScorer (and maybe others?) still call score() when scores are not needed. While this does not make results wrong, it is sub-optimal since it still forces to decode norms while they are not needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6427) BitSet fixes - assert on presence of 'ghost bits' and others
[ https://issues.apache.org/jira/browse/LUCENE-6427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518852#comment-14518852 ] Luc Vanlerberghe commented on LUCENE-6427: -- Thanks for committing! Actually, just mentioning This closes #142 in the commit message should be sufficient to trigger an ASF bot to close the pull request: (Just like my mentioning LUCENE-6427 in the pull request title triggered an automatic comment in the jira issue) See https://wiki.apache.org/lucene-java/BensonMarguliesGitWorkflow I'll close it manually, no problem. BitSet fixes - assert on presence of 'ghost bits' and others Key: LUCENE-6427 URL: https://issues.apache.org/jira/browse/LUCENE-6427 Project: Lucene - Core Issue Type: Bug Components: core/other Reporter: Luc Vanlerberghe Fix For: Trunk, 5.2 Fixes after reviewing org.apache.lucene.util.FixedBitSet, LongBitSet and corresponding tests: * Some methods rely on the fact that no bits are set after numBits (what I call 'ghost' bits here). ** cardinality, nextSetBit, intersects and others may yield wrong results ** If ghost bits are present, they may become visible after ensureCapacity is called. ** The tests deliberately create bitsets with ghost bits, but then do not detect these failures * FixedBitSet.cardinality scans the complete backing array, even if only numWords are in use -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[GitHub] lucene-solr pull request: Lucene 6427 bit set fixes
Github user LucVL closed the pull request at: https://github.com/apache/lucene-solr/pull/142 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. --- - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: [CI] Lucene 5x Linux Java8 64 Test Only - Build # 44439 - Failure!
I'll dig. Mike McCandless On Tue, Apr 28, 2015 at 8:24 PM, bu...@elastic.co wrote: *BUILD FAILURE* Build URL http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/ Project:lucene_linux_java8_64_test_only Randomization: JDK8,local,heap[1024m],-server +UseSerialGC +UseCompressedOops +AggressiveOpts,sec manager on Date of build:Wed, 29 Apr 2015 02:17:00 +0200 Build duration:7 min 12 sec *CHANGES* No Changes *BUILD ARTIFACTS* checkout/lucene/build/codecs/test/temp/junit4-J0-20150429_022315_738.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J0-20150429_022315_738.events checkout/lucene/build/codecs/test/temp/junit4-J1-20150429_022315_738.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J1-20150429_022315_738.events checkout/lucene/build/codecs/test/temp/junit4-J2-20150429_022315_738.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J2-20150429_022315_738.events checkout/lucene/build/codecs/test/temp/junit4-J3-20150429_022315_738.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J3-20150429_022315_738.events checkout/lucene/build/codecs/test/temp/junit4-J4-20150429_022315_739.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J4-20150429_022315_739.events checkout/lucene/build/codecs/test/temp/junit4-J5-20150429_022315_739.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J5-20150429_022315_739.events checkout/lucene/build/codecs/test/temp/junit4-J6-20150429_022315_739.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J6-20150429_022315_739.events checkout/lucene/build/codecs/test/temp/junit4-J7-20150429_022315_739.events http://build-eu-00.elastic.co/job/lucene_linux_java8_64_test_only/44439/artifact/checkout/lucene/build/codecs/test/temp/junit4-J7-20150429_022315_739.events *FAILED JUNIT TESTS* Name: org.apache.lucene.codecs.autoprefix Failed: 1 test(s), Passed: 7 test(s), Skipped: 0 test(s), Total: 8 test(s) *Failed: org.apache.lucene.codecs.autoprefix.TestAutoPrefixTerms.testBasicNumericRanges * *CONSOLE OUTPUT* [...truncated 7757 lines...] [junit4] [junit4] [junit4] JVM J0: 0.84 .. 50.26 = 49.42s [junit4] JVM J1: 0.85 .. 40.58 = 39.73s [junit4] JVM J2: 1.07 .. 43.91 = 42.84s [junit4] JVM J3: 0.84 .. 30.02 = 29.17s [junit4] JVM J4: 0.84 .. 43.20 = 42.36s [junit4] JVM J5: 0.84 .. 35.77 = 34.93s [junit4] JVM J6: 0.83 .. 40.51 = 39.68s [junit4] JVM J7: 0.84 .. 40.94 = 40.10s [junit4] Execution time total: 50 seconds [junit4] Tests summary: 21 suites, 680 tests, 1 failure, 33 ignored (33 assumptions) BUILD FAILED /home/jenkins/workspace/lucene_linux_java8_64_test_only/checkout/lucene/build.xml:466: The following error occurred while executing this line: /home/jenkins/workspace/lucene_linux_java8_64_test_only/checkout/lucene/common-build.xml:2229: The following error occurred while executing this line: /home/jenkins/workspace/lucene_linux_java8_64_test_only/checkout/lucene/module-build.xml:58: The following error occurred while executing this line: /home/jenkins/workspace/lucene_linux_java8_64_test_only/checkout/lucene/common-build.xml:1433: The following error occurred while executing this line: /home/jenkins/workspace/lucene_linux_java8_64_test_only/checkout/lucene/common-build.xml:991: There were test failures: 21 suites, 680 tests, 1 failure, 33 ignored (33 assumptions) Total time: 7 minutes 2 seconds Build step 'Invoke Ant' marked build as failure Archiving artifacts Recording test results [description-setter] Description set: JDK8,local,heap[1024m],-server +UseSerialGC +UseCompressedOops +AggressiveOpts,sec manager on Email was triggered for: Failure - 1st Trigger Failure - Any was overridden by another trigger and will not send an email. Trigger Failure - Still was overridden by another trigger and will not send an email. Sending email for trigger: Failure - 1st
[jira] [Resolved] (LUCENE-6328) Make Filter.clone and Filter.setBoost throw an UnsupportedOperationException
[ https://issues.apache.org/jira/browse/LUCENE-6328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-6328. -- Resolution: Won't Fix Fix Version/s: (was: 5.2) (was: Trunk) Make Filter.clone and Filter.setBoost throw an UnsupportedOperationException Key: LUCENE-6328 URL: https://issues.apache.org/jira/browse/LUCENE-6328 Project: Lucene - Core Issue Type: Bug Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6328.patch, queryextendsfilter-20150307.patch, queryextendsfilter-20150307b.patch The rewrite process uses a combination of calls to clone() and setBoost(boost) in order to rewrite queries. This is a bit weird for filters given that they were not originally designed to care about scoring. Using a filter directly as a query fails unit tests today since filters do not pass the QueryUtils checks: it is expected that cloning and changing the boost results in an instance which is unequal. However existing filters do not take into account the getBoost() parameter inherited from Query so this test fails. I think it would be less error-prone to throw an UnsupportedOperationException for clone() and setBoost() on filters and disable the check in QueryUtils for filters. In order to keep rewriting working, filters could rewrite to a CSQ around themselves so that clone() and setBoost() would be called on the CSQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-6213) Add test for IndexFormatTooOldException if a commit has a 3.x segment
[ https://issues.apache.org/jira/browse/LUCENE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518752#comment-14518752 ] Ryan Ernst edited comment on LUCENE-6213 at 4/29/15 5:18 AM: - bq. i think that even if there is no official M release, the smoketester for M+1 should fail if a test for (M)x-with-(M-1)x-segments can not be found. I disagree. We shouldn't fail when there is nothing we can do. The backcompat indexes are created with official releases, so we should not fail until there is an official release with which to build the backcompat index (whether that be a pure index or mixed). was (Author: rjernst): i think that even if there is no official M release, the smoketester for M+1 should fail if a test for (M)x-with-(M-1)x-segments can not be found. I disagree. We shouldn't fail when there is nothing we can do. The backcompat indexes are created with official releases, so we should not fail until there is an official release with which to build the backcompat index (whether that be a pure index or mixed). Add test for IndexFormatTooOldException if a commit has a 3.x segment - Key: LUCENE-6213 URL: https://issues.apache.org/jira/browse/LUCENE-6213 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Ryan Ernst Attachments: LUCENE-6213.patch, LUCENE-6213.patch, LUCENE-6213.patch, unsupported.4x-with-3x-segments.zip We should add a 4.x index (4.x commit) with some 3.x segment(s) to our backwards tests. I don't think we throw IndexFormatTooOldException correctly in this case. I think instead the user will get a confusing SPI error about a missing codec Lucene3x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Linux (32bit/jdk1.8.0_60-ea-b12) - Build # 12324 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Linux/12324/ Java: 32bit/jdk1.8.0_60-ea-b12 -client -XX:+UseConcMarkSweepGC 1 tests failed. FAILED: org.apache.solr.search.facet.TestJsonFacets.testComplex Error Message: mismatch: 'civic'!='a' @ facets/makes/buckets/[0]/models/buckets/[1]/val Stack Trace: java.lang.RuntimeException: mismatch: 'civic'!='a' @ facets/makes/buckets/[0]/models/buckets/[1]/val at __randomizedtesting.SeedInfo.seed([7E1528D4C05EEF99:9FCA2D48EC10A3FA]:0) at org.apache.solr.SolrTestCaseHS.matchJSON(SolrTestCaseHS.java:160) at org.apache.solr.SolrTestCaseHS.assertJQ(SolrTestCaseHS.java:142) at org.apache.solr.SolrTestCaseHS$Client.testJQ(SolrTestCaseHS.java:288) at org.apache.solr.search.facet.TestJsonFacets.testComplex(TestJsonFacets.java:157) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 9919 lines...] [junit4] Suite:
[jira] [Commented] (LUCENE-6213) Add test for IndexFormatTooOldException if a commit has a 3.x segment
[ https://issues.apache.org/jira/browse/LUCENE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14518752#comment-14518752 ] Ryan Ernst commented on LUCENE-6213: i think that even if there is no official M release, the smoketester for M+1 should fail if a test for (M)x-with-(M-1)x-segments can not be found. I disagree. We shouldn't fail when there is nothing we can do. The backcompat indexes are created with official releases, so we should not fail until there is an official release with which to build the backcompat index (whether that be a pure index or mixed). Add test for IndexFormatTooOldException if a commit has a 3.x segment - Key: LUCENE-6213 URL: https://issues.apache.org/jira/browse/LUCENE-6213 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Ryan Ernst Attachments: LUCENE-6213.patch, LUCENE-6213.patch, LUCENE-6213.patch, unsupported.4x-with-3x-segments.zip We should add a 4.x index (4.x commit) with some 3.x segment(s) to our backwards tests. I don't think we throw IndexFormatTooOldException correctly in this case. I think instead the user will get a confusing SPI error about a missing codec Lucene3x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Fwd: JEP 248: Make G1 the Default Garbage Collector
I think this may be of importance to us since we specifically don't like G1GC? :) D. -- Forwarded message -- From: mark.reinh...@oracle.com Date: Wed, Apr 29, 2015 at 12:02 AM Subject: JEP 248: Make G1 the Default Garbage Collector To: stefan.johans...@oracle.com Cc: hotspot-...@openjdk.java.net New JEP Candidate: http://openjdk.java.net/jeps/248 - Mark - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
[ https://issues.apache.org/jira/browse/SOLR-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shai Erera updated SOLR-7486: - Attachment: SOLR-7486.patch Trivial patch. This only affects 5x as shutdown() has been removed in trunk already. HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 Attachments: SOLR-7486.patch HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
[
https://issues.apache.org/jira/browse/SOLR-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519752#comment-14519752
]
Hoss Man commented on SOLR-7488:
Here's the code in question...
{noformat}
897383 koji // get FastVectorHighlighter instance out of the
processing loop
897383 koji FastVectorHighlighter fvh = new FastVectorHighlighter(
897383 koji // FVH cannot process hl.usePhraseHighlighter
parameter per-field basis
897383 koji params.getBool(
HighlightParams.USE_PHRASE_HIGHLIGHTER, true ),
897383 koji // FVH cannot process hl.requireFieldMatch parameter
per-field basis
990301 koji params.getBool( HighlightParams.FIELD_MATCH, false )
);
1555369 rmuir
fvh.setPhraseLimit(params.getInt(HighlightParams.PHRASE_LIMIT,
SolrHighlighter.DEFAULT_PHRASE_LIMIT));
1167008 rmuir FieldQuery fieldQuery = fvh.getFieldQuery( query,
searcher.getIndexReader() );
639490 klaas
639490 klaas // Highlight each document
639490 klaas DocIterator iterator = docs.iterator();
639490 klaas for (int i = 0; i docs.size(); i++) {
897383 koji int docId = iterator.nextDoc();
897383 koji Document doc = searcher.doc(docId, fset);
897383 koji NamedList docSummaries = new SimpleOrderedMap();
897383 koji for (String fieldName : fieldNames) {
897383 koji fieldName = fieldName.trim();
897383 koji if( useFastVectorHighlighter( params, schema,
fieldName ) )
990301 koji doHighlightingByFastVectorHighlighter( fvh,
fieldQuery, req, docSummaries, docId, doc, fieldName );
897383 koji else
897383 koji doHighlightingByHighlighter( query, req,
docSummaries, docId, doc, fieldName );
897383 koji }
897383 koji String printId = schema.printableUniqueKey(doc);
897383 koji fragments.add(printId == null ? null : printId,
docSummaries);
897383 koji }
{noformat}
It looks like the bug may have been introduced by r1167008 for LUCENE-1889:
pass the reader to FVH so it can rewrite multitermqueries ... because
DefaultSolrHighlighter had previously optimized the FVH call out of the loop,
this caused the rewriting to happen even if the user had said they didn't want
FVH.
[~koji] [~dsmiley] - can one of you sanity check me here? (I don't
understand the surround parser and/or highlighter well enough to try and write
a test case for this)
suspicious FVH init code in DefaultSolrHighlighter even when FVH should not
be used
---
Key: SOLR-7488
URL: https://issues.apache.org/jira/browse/SOLR-7488
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: Hoss Man
Rich Hume reported gettting errors from FastVectorHighlighter, evidently
while using the the surround query parser, even though he was not trying to
useFastVectorHighlighter
my naive reading of the code leads me to believe that DefaultSolrHighlighter
is incorrectly attempting to initialize a FVH instance even when it shouldn't
be -- which appears to cause failures in cases where the query in use is not
something that can be handled by the FVH.
Not sure how to reproduce at the moment -- but the code smells fishy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-5.x-Windows (32bit/jdk1.7.0_80) - Build # 4624 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/4624/
Java: 32bit/jdk1.7.0_80 -client -XX:+UseSerialGC
1 tests failed.
FAILED: org.apache.solr.cloud.TestCloudPivotFacet.test
Error Message:
init query failed:
{main(facet=truefacet.pivot=%7B%21stats%3Dst0%7Dpivot_td1%2Cpivot_lfacet.pivot=%7B%21stats%3Dst3%7Ddense_pivot_ti%2Cpivot_y_s%2Cpivot_dfacet.limit=17facet.offset=1facet.missing=true),extra(rows=0q=id%3A%5B*+TO+425%5Dfq=id%3A%5B*+TO+458%5D_test_miss=true)}:
No live SolrServers available to handle this
request:[http://127.0.0.1:54698/_awa/pl/collection1,
http://127.0.0.1:54743/_awa/pl/collection1,
http://127.0.0.1:54731/_awa/pl/collection1,
http://127.0.0.1:54722/_awa/pl/collection1]
Stack Trace:
java.lang.RuntimeException: init query failed:
{main(facet=truefacet.pivot=%7B%21stats%3Dst0%7Dpivot_td1%2Cpivot_lfacet.pivot=%7B%21stats%3Dst3%7Ddense_pivot_ti%2Cpivot_y_s%2Cpivot_dfacet.limit=17facet.offset=1facet.missing=true),extra(rows=0q=id%3A%5B*+TO+425%5Dfq=id%3A%5B*+TO+458%5D_test_miss=true)}:
No live SolrServers available to handle this
request:[http://127.0.0.1:54698/_awa/pl/collection1,
http://127.0.0.1:54743/_awa/pl/collection1,
http://127.0.0.1:54731/_awa/pl/collection1,
http://127.0.0.1:54722/_awa/pl/collection1]
at
__randomizedtesting.SeedInfo.seed([B598C76F5791254E:3DCCF8B5F96D48B6]:0)
at
org.apache.solr.cloud.TestCloudPivotFacet.assertPivotCountsAreCorrect(TestCloudPivotFacet.java:254)
at
org.apache.solr.cloud.TestCloudPivotFacet.test(TestCloudPivotFacet.java:228)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960)
at
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39)
at
[jira] [Commented] (LUCENE-6453) Specialize SpanPositionQueue similar to DisiPriorityQueue
[ https://issues.apache.org/jira/browse/LUCENE-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519620#comment-14519620 ] Paul Elschot commented on LUCENE-6453: -- This should also be used in NearSpansUnordered, but the current patch does not do that. Specialize SpanPositionQueue similar to DisiPriorityQueue - Key: LUCENE-6453 URL: https://issues.apache.org/jira/browse/LUCENE-6453 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Fix For: Trunk, 5.x Attachments: LUCENE-6453.patch, LUCENE-6453.patch Inline the position comparison function -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
[
https://issues.apache.org/jira/browse/SOLR-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519733#comment-14519733
]
Hoss Man commented on SOLR-7488:
Details from initial user report...
{noformat}
After we upgraded Solr from 4.5.1 to 4.10.4, we started seeing the following
UnsupportedOperationException logged
repeatedly. We do not have highlighting configured to
useFastVectorHighlighter. The logged stack trace has given
me little to go on. I was hoping this is a problem others have seen (but my
searching has yielded nothing).
Thanks
Rich
null:java.lang.UnsupportedOperationException
at
org.apache.lucene.queryparser.surround.query.RewriteQuery.clone(RewriteQuery.java:80)
at
org.apache.lucene.queryparser.surround.query.RewriteQuery.clone(RewriteQuery.java:23)
at
org.apache.lucene.search.vectorhighlight.FieldQuery.applyParentBoost(FieldQuery.java:156)
at
org.apache.lucene.search.vectorhighlight.FieldQuery.flatten(FieldQuery.java:96)
at
org.apache.lucene.search.vectorhighlight.FieldQuery.flatten(FieldQuery.java:96)
at
org.apache.lucene.search.vectorhighlight.FieldQuery.init(FieldQuery.java:67)
at
org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFieldQuery(FastVectorHighlighter.java:98)
at
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:401)
at
org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:144)
at
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1976)
{noformat}
suspicious FVH init code in DefaultSolrHighlighter even when FVH should not
be used
---
Key: SOLR-7488
URL: https://issues.apache.org/jira/browse/SOLR-7488
Project: Solr
Issue Type: Bug
Affects Versions: 4.10
Reporter: Hoss Man
Rich Hume reported gettting errors from FastVectorHighlighter, evidently
while using the the surround query parser, even though he was not trying to
useFastVectorHighlighter
my naive reading of the code leads me to believe that DefaultSolrHighlighter
is incorrectly attempting to initialize a FVH instance even when it shouldn't
be -- which appears to cause failures in cases where the query in use is not
something that can be handled by the FVH.
Not sure how to reproduce at the moment -- but the code smells fishy.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-6459) [suggest] Query Interface for suggest API
Areek Zillur created LUCENE-6459:
Summary: [suggest] Query Interface for suggest API
Key: LUCENE-6459
URL: https://issues.apache.org/jira/browse/LUCENE-6459
Project: Lucene - Core
Issue Type: New Feature
Components: core/search
Affects Versions: 5.1
Reporter: Areek Zillur
Assignee: Areek Zillur
Fix For: Trunk, 5.x, 5.1
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit distance
of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
[ https://issues.apache.org/jira/browse/SOLR-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519695#comment-14519695 ] Tomás Fernández Löbbe commented on SOLR-7486: - In general, I think the way it is now is better for back compatibility when deprecating. If you have extended HttpSolrClient (before the addition of close()) and overrode shutdown(), then your code may stop working. That said, in this specific case, you are probably also calling super.shutdown(), because otherwise the httpClients are not closed, but we can only be sure about that in this case because those are private fields. bq. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. If you are extending now, you should extend close(), and make sure you call super.close() at some point. That will work with 5 and trunk. HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 Attachments: SOLR-7486.patch HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
[ https://issues.apache.org/jira/browse/SOLR-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519695#comment-14519695 ] Tomás Fernández Löbbe edited comment on SOLR-7486 at 4/29/15 4:43 PM: -- In general, I think the way it is now is better for back compatibility when deprecating. If you have extended HttpSolrClient (before the addition of close()) and overrode shutdown(), then your code could stop working with your change. That said, in this specific case, you are probably also calling super.shutdown(), because otherwise the httpClients are not closed, but we can only be sure about that in this case because those are private fields. bq. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. If you are extending now, you should extend close(), and make sure you call super.close() at some point. That will work with 5 and trunk. was (Author: tomasflobbe): In general, I think the way it is now is better for back compatibility when deprecating. If you have extended HttpSolrClient (before the addition of close()) and overrode shutdown(), then your code may stop working. That said, in this specific case, you are probably also calling super.shutdown(), because otherwise the httpClients are not closed, but we can only be sure about that in this case because those are private fields. bq. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. If you are extending now, you should extend close(), and make sure you call super.close() at some point. That will work with 5 and trunk. HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 Attachments: SOLR-7486.patch HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6213) Add test for IndexFormatTooOldException if a commit has a 3.x segment
[
https://issues.apache.org/jira/browse/LUCENE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519646#comment-14519646
]
Robert Muir commented on LUCENE-6213:
-
{quote}
I disagree. We shouldn't fail when there is nothing we can do.
{quote}
Yeah, failing jenkins over and over for a potentially long period of time is
not a solution here.
We can release version X without tests that it can read X-1. In general, test
coverage is never perfect.
Add test for IndexFormatTooOldException if a commit has a 3.x segment
-
Key: LUCENE-6213
URL: https://issues.apache.org/jira/browse/LUCENE-6213
Project: Lucene - Core
Issue Type: Bug
Reporter: Robert Muir
Assignee: Ryan Ernst
Attachments: LUCENE-6213.patch, LUCENE-6213.patch, LUCENE-6213.patch,
unsupported.4x-with-3x-segments.zip
We should add a 4.x index (4.x commit) with some 3.x segment(s) to our
backwards tests.
I don't think we throw IndexFormatTooOldException correctly in this case. I
think instead the user will get a confusing SPI error about a missing codec
Lucene3x.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6213) Add test for IndexFormatTooOldException if a commit has a 3.x segment
[ https://issues.apache.org/jira/browse/LUCENE-6213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519640#comment-14519640 ] Hoss Man commented on LUCENE-6213: -- bq. We shouldn't fail when there is nothing we can do. -0 In my opinion you are looking at the purpose of the smoke tester backwards: the smoke tester shouldn't _pass_ unless the candidate it's smoking is ready for release -- if we know that there is no previous index format that is being adequately tested in the candidate build, then the candidate is, by definition, not ready for release. Add test for IndexFormatTooOldException if a commit has a 3.x segment - Key: LUCENE-6213 URL: https://issues.apache.org/jira/browse/LUCENE-6213 Project: Lucene - Core Issue Type: Bug Reporter: Robert Muir Assignee: Ryan Ernst Attachments: LUCENE-6213.patch, LUCENE-6213.patch, LUCENE-6213.patch, unsupported.4x-with-3x-segments.zip We should add a 4.x index (4.x commit) with some 3.x segment(s) to our backwards tests. I don't think we throw IndexFormatTooOldException correctly in this case. I think instead the user will get a confusing SPI error about a missing codec Lucene3x. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6459) [suggest] Query Interface for suggest API
[
https://issues.apache.org/jira/browse/LUCENE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Areek Zillur updated LUCENE-6459:
-
Description:
Related Issue: [NRTSuggester|https://issues.apache.org/jira/browse/LUCENE-6339]
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit distance
of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
was:
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit distance
of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
[suggest] Query Interface for suggest API
-
Key: LUCENE-6459
URL: https://issues.apache.org/jira/browse/LUCENE-6459
Project: Lucene - Core
Issue Type: New Feature
Components: core/search
Affects Versions: 5.1
Reporter: Areek Zillur
Assignee: Areek
[jira] [Updated] (LUCENE-6459) [suggest] Query Interface for suggest API
[
https://issues.apache.org/jira/browse/LUCENE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Areek Zillur updated LUCENE-6459:
-
Attachment: LUCENE-6459.patch
Initial patch. It would be awesome to get some feedback on this!
[suggest] Query Interface for suggest API
-
Key: LUCENE-6459
URL: https://issues.apache.org/jira/browse/LUCENE-6459
Project: Lucene - Core
Issue Type: New Feature
Components: core/search
Affects Versions: 5.1
Reporter: Areek Zillur
Assignee: Areek Zillur
Fix For: Trunk, 5.x, 5.1
Attachments: LUCENE-6459.patch
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit
distance of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7487) check-example-lucene-match-version is looking in the wrong place - luceneMatchVersion incorrect in 5.1 sample configs
Hoss Man created SOLR-7487:
--
Summary: check-example-lucene-match-version is looking in the
wrong place - luceneMatchVersion incorrect in 5.1 sample configs
Key: SOLR-7487
URL: https://issues.apache.org/jira/browse/SOLR-7487
Project: Solr
Issue Type: Bug
Affects Versions: 5.1
Reporter: Hoss Man
Priority: Blocker
Fix For: 5.2
As noted by Scott Dawson on the mailing list, the luceneMatchVersion in the 5.1
sample configs all still lists 5.0.0.
The root cause seems to be because the check-example-lucene-match-version task
in solr/build.xml is looking in the wrong place -- it's still scaning for
instances of luceneMatchVersion in the {{example}} directory instead of the
{{server/solr/configset}}
TODO:
* fix the luceneMatchVersion value in all sample configsets on 5x
* update the check to look in the correct directory
* update the check to fail to be smarter know that we have a more predictable
directory structure
** fail if no subdirs found
** fail if any subdir doesn't contain conf/solrconfig.xml
** fail if any conf/solrconfig.xml doesn't contain a luceneMatchVersion
** fail if any luceneMatchVersion doesn't have the expected value
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
Hoss Man created SOLR-7488: -- Summary: suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used Key: SOLR-7488 URL: https://issues.apache.org/jira/browse/SOLR-7488 Project: Solr Issue Type: Bug Affects Versions: 4.10 Reporter: Hoss Man Rich Hume reported gettting errors from FastVectorHighlighter, evidently while using the the surround query parser, even though he was not trying to useFastVectorHighlighter my naive reading of the code leads me to believe that DefaultSolrHighlighter is incorrectly attempting to initialize a FVH instance even when it shouldn't be -- which appears to cause failures in cases where the query in use is not something that can be handled by the FVH. Not sure how to reproduce at the moment -- but the code smells fishy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7275) Pluggable authorization module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519638#comment-14519638
]
Anshum Gupta commented on SOLR-7275:
From http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html:
401:
The client MAY repeat the request with a suitable Authorization header field
(section 14.8). If the request already included Authorization credentials, then
the 401 response indicates that authorization has been refused for those
credentials.
403:
The server understood the request, but is refusing to fulfill it. Authorization
will not help and the request SHOULD NOT be repeated. If the request method was
not HEAD and the server wishes to make public why the request has not been
fulfilled, it SHOULD describe the reason for the refusal in the entity.
As per what I undersand, I think 401 still makes more sense in this case.
Pluggable authorization module in Solr
--
Key: SOLR-7275
URL: https://issues.apache.org/jira/browse/SOLR-7275
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Attachments: SOLR-7275.patch, SOLR-7275.patch, SOLR-7275.patch
Solr needs an interface that makes it easy for different authorization
systems to be plugged into it. Here's what I plan on doing:
Define an interface {{SolrAuthorizationPlugin}} with one single method
{{isAuthorized}}. This would take in a {{SolrRequestContext}} object and
return an {{SolrAuthorizationResponse}} object. The object as of now would
only contain a single boolean value but in the future could contain more
information e.g. ACL for document filtering etc.
The reason why we need a context object is so that the plugin doesn't need to
understand Solr's capabilities e.g. how to extract the name of the collection
or other information from the incoming request as there are multiple ways to
specify the target collection for a request. Similarly request type can be
specified by {{qt}} or {{/handler_name}}.
Flow:
Request - SolrDispatchFilter - isAuthorized(context) - Process/Return.
{code}
public interface SolrAuthorizationPlugin {
public SolrAuthorizationResponse isAuthorized(SolrRequestContext context);
}
{code}
{code}
public class SolrRequestContext {
UserInfo; // Will contain user context from the authentication layer.
HTTPRequest request;
Enum OperationType; // Correlated with user roles.
String[] CollectionsAccessed;
String[] FieldsAccessed;
String Resource;
}
{code}
{code}
public class SolrAuthorizationResponse {
boolean authorized;
public boolean isAuthorized();
}
{code}
User Roles:
* Admin
* Collection Level:
* Query
* Update
* Admin
Using this framework, an implementation could be written for specific
security systems e.g. Apache Ranger or Sentry. It would keep all the security
system specific code out of Solr.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7275) Pluggable authorization module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519667#comment-14519667
]
Ishan Chattopadhyaya commented on SOLR-7275:
From the same source, I can see this for 401:
bq. The request requires user authentication. The response MUST include a
WWW-Authenticate header field (section 14.47) containing a challenge applicable
to the requested resource.
So, that means a 401 response must also contain a WWW-Authenticate header with
a challenge, which seems inappropriate in this situation. Hence, I still feel
403 is more appropriate.
Pluggable authorization module in Solr
--
Key: SOLR-7275
URL: https://issues.apache.org/jira/browse/SOLR-7275
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Attachments: SOLR-7275.patch, SOLR-7275.patch, SOLR-7275.patch
Solr needs an interface that makes it easy for different authorization
systems to be plugged into it. Here's what I plan on doing:
Define an interface {{SolrAuthorizationPlugin}} with one single method
{{isAuthorized}}. This would take in a {{SolrRequestContext}} object and
return an {{SolrAuthorizationResponse}} object. The object as of now would
only contain a single boolean value but in the future could contain more
information e.g. ACL for document filtering etc.
The reason why we need a context object is so that the plugin doesn't need to
understand Solr's capabilities e.g. how to extract the name of the collection
or other information from the incoming request as there are multiple ways to
specify the target collection for a request. Similarly request type can be
specified by {{qt}} or {{/handler_name}}.
Flow:
Request - SolrDispatchFilter - isAuthorized(context) - Process/Return.
{code}
public interface SolrAuthorizationPlugin {
public SolrAuthorizationResponse isAuthorized(SolrRequestContext context);
}
{code}
{code}
public class SolrRequestContext {
UserInfo; // Will contain user context from the authentication layer.
HTTPRequest request;
Enum OperationType; // Correlated with user roles.
String[] CollectionsAccessed;
String[] FieldsAccessed;
String Resource;
}
{code}
{code}
public class SolrAuthorizationResponse {
boolean authorized;
public boolean isAuthorized();
}
{code}
User Roles:
* Admin
* Collection Level:
* Query
* Update
* Admin
Using this framework, an implementation could be written for specific
security systems e.g. Apache Ranger or Sentry. It would keep all the security
system specific code out of Solr.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6453) Specialize SpanPositionQueue similar to DisiPriorityQueue
[ https://issues.apache.org/jira/browse/LUCENE-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Paul Elschot updated LUCENE-6453: - Attachment: LUCENE-6453.patch Patch of 29 April 2015. Use generic SpanPositionQueue in SpanOr and in NearSpansUnordered. Specialize SpanPositionQueue similar to DisiPriorityQueue - Key: LUCENE-6453 URL: https://issues.apache.org/jira/browse/LUCENE-6453 Project: Lucene - Core Issue Type: Improvement Components: core/search Reporter: Paul Elschot Priority: Minor Fix For: Trunk, 5.x Attachments: LUCENE-6453.patch, LUCENE-6453.patch, LUCENE-6453.patch Inline the position comparison function -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 3032 - Still Failing
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3032/ 1 tests failed. FAILED: org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds Error Message: soft529 wasn't fast enough Stack Trace: java.lang.AssertionError: soft529 wasn't fast enough at __randomizedtesting.SeedInfo.seed([D7BD946C5EB198DA:86696DECEFC2A87D]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNotNull(Assert.java:526) at org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds(SoftAutoCommitTest.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 9412 lines...] [junit4] Suite: org.apache.solr.update.SoftAutoCommitTest [junit4] 2 Creating dataDir:
[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520125#comment-14520125 ] Per Steffensen commented on SOLR-6810: -- Happy to see the activity here. Currently I am very busy, so I do not have the time to follow the work closely. Hope I will soon be able to. Faster searching limited but high rows across many shards all with many hits Key: SOLR-6810 URL: https://issues.apache.org/jira/browse/SOLR-6810 Project: Solr Issue Type: Improvement Components: search Reporter: Per Steffensen Assignee: Shalin Shekhar Mangar Labels: distributed_search, performance Attachments: SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, branch_5x_rev1645549.patch Searching limited but high rows across many shards all with many hits is slow E.g. * Query from outside client: q=somethingrows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=somethingrows=1000fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * limited but high rows means 1000 in the example above * many shards means 200-1000 in our case * all with many hits means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. Profiling show that the problem is that it takes lots of time to access the store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched. There is really no good reason to ever read information from store for more than the overall top-1000 documents, that has to be returned to the client. For further detail see mail-thread Slow searching limited but high rows across many shards all with high hits started 13/11-2014 on dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
[ https://issues.apache.org/jira/browse/SOLR-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man resolved SOLR-7488. Resolution: Fixed Fix Version/s: 5.2 Trunk Assignee: David Smiley Doh! .. thanks David. i must have been looking at the 5_1 branch not 5x. suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used --- Key: SOLR-7488 URL: https://issues.apache.org/jira/browse/SOLR-7488 Project: Solr Issue Type: Bug Affects Versions: 4.10 Reporter: Hoss Man Assignee: David Smiley Fix For: Trunk, 5.2 Rich Hume reported gettting errors from FastVectorHighlighter, evidently while using the the surround query parser, even though he was not trying to useFastVectorHighlighter my naive reading of the code leads me to believe that DefaultSolrHighlighter is incorrectly attempting to initialize a FVH instance even when it shouldn't be -- which appears to cause failures in cases where the query in use is not something that can be handled by the FVH. Not sure how to reproduce at the moment -- but the code smells fishy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-6459) [suggest] Query Interface for suggest API
[
https://issues.apache.org/jira/browse/LUCENE-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Areek Zillur updated LUCENE-6459:
-
Description:
Related Issue: [NRTSuggester|https://issues.apache.org/jira/browse/LUCENE-6339]
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit distance
of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
When used with {{FuzzyCompletionQuery}},
{{suggestion_weight + (global_maximum_weight * (context_boost + fuzzy_boost))}}
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
was:
Related Issue: [NRTSuggester|https://issues.apache.org/jira/browse/LUCENE-6339]
h3. Suggest API:
{code}
SuggestIndexSearcher searcher = new SuggestIndexSearcher(reader);
CompletionQuery query = ...
TopSuggestDocs suggest = searcher.suggest(query, num);
{code}
h3. Query Interface
h4. PrefixCompletionQuery
Return documents with values that match the prefix of an analyzed term text
Documents are sorted according to their suggest field weight.
{code}
PrefixCompletionQuery(Analyzer analyzer, Term term)
{code}
h4. RegexCompletionQuery
Return documents with values that match the prefix of a regular expression
Documents are sorted according to their suggest field weight.
{code}
RegexCompletionQuery(Term term)
{code}
h4. FuzzyCompletionQuery
Return documents with values that has prefixes within a specified edit distance
of an analyzed term text.
Documents are ‘boosted’ by the number of matching prefix letters of the
suggestion with respect to the original term text.
{code}
FuzzyCompletionQuery(Analyzer analyzer, Term term)
{code}
h5. Scoring
{{suggestion_weight + (global_maximum_weight * boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{boost}} are all
integers.
{{boost = # of prefix characters matched}}
h4. ContextQuery
Return documents that match a {{CompletionQuery}} filtered and/or boosted by
provided context(s).
{code}
ContextQuery(CompletionQuery query)
contextQuery.addContext(CharSequence context, int boost, boolean exact)
{code}
*NOTE:* {{ContextQuery}} should be used with {{ContextSuggestField}} to query
suggestions boosted and/or filtered by contexts
h5. Scoring
{{suggestion_weight + (global_maximum_weight * context_boost)}}
where {{suggestion_weight}}, {{global_maximum_weight}} and {{context_boost}}
are all integers
h3. Context Suggest Field
To use {{ContextQuery}}, use {{ContextSuggestField}} instead of
{{SuggestField}}. Any {{CompletionQuery}} can be used with
{{ContextSuggestField}}, the default behaviour is to return suggestions from
*all* contexts. {{Context}} for every completion hit can be accessed through
{{SuggestScoreDoc#context}}.
{code}
ContextSuggestField(String name, CollectionCharSequence contexts, String
value, int weight)
{code}
[suggest] Query Interface for suggest API
-
Key: LUCENE-6459
URL:
[jira] [Updated] (SOLR-7490) Update by query feature
[
https://issues.apache.org/jira/browse/SOLR-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Praneeth updated SOLR-7490:
---
Priority: Minor (was: Major)
Update by query feature
---
Key: SOLR-7490
URL: https://issues.apache.org/jira/browse/SOLR-7490
Project: Solr
Issue Type: New Feature
Reporter: Praneeth
Priority: Minor
An update feature similar to the {{deleteByQuery}} would be very useful. Say,
the user wants to update a field of all documents in the index that match a
given criteria. I have encountered this use case in my project and it looks
like it could be a useful first class solr/lucene feature. I want to check if
this is something we would want to support in coming releases of Solr and
Lucene, are there scenarios that will prevent us from doing this,
feasibility, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7490) Update by query feature
[
https://issues.apache.org/jira/browse/SOLR-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Praneeth updated SOLR-7490:
---
Description: An update feature similar to the {{deleteByQuery}} would be
very useful. Say, the user wants to update a field of all documents in the
index that match a given criteria. I have encountered this use case in my
project and it looks like it could be a useful first class solr/lucene feature.
I want to check if this is something we would want to support in coming
releases of Solr and Lucene, are there scenarios that will prevent us from
doing this, feasibility, etc. (was: An update feature similar to the
{{deleteByQuery}} would be very useful. Say, the user wants to update a field
of all documents in the index that match a given criteria. I have encountered
this use case in my project and it looks like it could be a useful first class
solr/lucene feature. I want to check if this is something we would want to
support in coming releases of Solr, are there scenarios that will prevent us
from doing this, feasibility, etc.)
Update by query feature
---
Key: SOLR-7490
URL: https://issues.apache.org/jira/browse/SOLR-7490
Project: Solr
Issue Type: New Feature
Reporter: Praneeth
An update feature similar to the {{deleteByQuery}} would be very useful. Say,
the user wants to update a field of all documents in the index that match a
given criteria. I have encountered this use case in my project and it looks
like it could be a useful first class solr/lucene feature. I want to check if
this is something we would want to support in coming releases of Solr and
Lucene, are there scenarios that will prevent us from doing this,
feasibility, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[
https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519895#comment-14519895
]
Noble Paul commented on SOLR-7484:
--
I have moved the entire functionality into a class called {{SolrCall}} which
represents a call to Solr
Refactor SolrDispatchFilter.doFilter(...) method
Key: SOLR-7484
URL: https://issues.apache.org/jira/browse/SOLR-7484
Project: Solr
Issue Type: Improvement
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Attachments: SOLR-7484.patch, SOLR-7484.patch
Currently almost everything that's done in SDF.doFilter() is sequential. We
should refactor it to clean up the code and make things easier to manage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[
https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519895#comment-14519895
]
Noble Paul edited comment on SOLR-7484 at 4/29/15 6:37 PM:
---
I have moved the entire functionality into a class called {{SolrCall}} which
represents a call to Solr. {{SolrDispatchFilter}} is now just reduced to a
proxy which just creates a {{SolrCall}} and passes the control over there
was (Author: noble.paul):
I have moved the entire functionality into a class called {{SolrCall}} which
represents a call to Solr
Refactor SolrDispatchFilter.doFilter(...) method
Key: SOLR-7484
URL: https://issues.apache.org/jira/browse/SOLR-7484
Project: Solr
Issue Type: Improvement
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Attachments: SOLR-7484.patch, SOLR-7484.patch
Currently almost everything that's done in SDF.doFilter() is sequential. We
should refactor it to clean up the code and make things easier to manage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
[ https://issues.apache.org/jira/browse/SOLR-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519806#comment-14519806 ] Shai Erera commented on SOLR-7486: -- This is the usual pattern when deprecating code - you make the deprecated code call the non-deprecated code (when possible). Also in this case users have to call super.shutdown() to close the client. In my case I need to provide my own HttpClient which I cannot save a reference to outside HttpSolrClient. So I pass my custom HttpClient to HttpSolrClient constructor. However, that makes HttpSolrClient determine my client isn't an internalClient, and therefore it doesn't close it on shutdown/close. I need to do it myself, only I now need to override both close() and shutdown(). With this change, and in the context of HttpSolrClient, I believe this patch is OK. It follows usual deprecation practice (at least that I've seen), it ensures the HttpClient is closed if you call either shutdown or close, it doesn't impact users who only override shutdown(), since as you mentioned, they need to call super.shutdown() already and when the day comes and we remove this deprecated API, we only need to delete the method, and not worry about copying code to close(). BTW, are we really going to carry this API 'till 6.0? If not, it's been deprecated in 5.0 and 5.1, and we can just remove it now ... HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 Attachments: SOLR-7486.patch HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7490) Update by query feature
Praneeth created SOLR-7490:
--
Summary: Update by query feature
Key: SOLR-7490
URL: https://issues.apache.org/jira/browse/SOLR-7490
Project: Solr
Issue Type: New Feature
Reporter: Praneeth
An update feature similar to the {{deleteByQuery}} would be very useful. Say,
the user wants to update a field of all documents in the index that match a
given criteria. I have encountered this use case in my project and it looks
like it could be a useful first class solr/lucene feature. I want to check if
this is something we would want to support in coming releases of Solr, are
there scenarios that will prevent us from doing this, feasibility, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-7484: - Attachment: SOLR-7484.patch Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7377) SOLR Streaming Expressions
[
https://issues.apache.org/jira/browse/SOLR-7377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Bernstein updated SOLR-7377:
-
Attachment: SOLR-7377.patch
New patch is an svn diff that handles all the moving around of files in svn.
SOLR Streaming Expressions
--
Key: SOLR-7377
URL: https://issues.apache.org/jira/browse/SOLR-7377
Project: Solr
Issue Type: Improvement
Components: clients - java
Reporter: Dennis Gove
Priority: Minor
Fix For: Trunk
Attachments: SOLR-7377.patch, SOLR-7377.patch, SOLR-7377.patch,
SOLR-7377.patch
It would be beneficial to add an expression-based interface to Streaming API
described in SOLR-7082. Right now that API requires streaming requests to
come in from clients as serialized bytecode of the streaming classes. The
suggestion here is to support string expressions which describe the streaming
operations the client wishes to perform.
{code:java}
search(collection1, q=*:*, fl=id,fieldA,fieldB, sort=fieldA asc)
{code}
With this syntax in mind, one can now express arbitrarily complex stream
queries with a single string.
{code:java}
// merge two distinct searches together on common fields
merge(
search(collection1, q=id:(0 3 4), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
search(collection2, q=id:(1 2), fl=id,a_s,a_i,a_f, sort=a_f asc, a_s
asc),
on=a_f asc, a_s asc)
// find top 20 unique records of a search
top(
n=20,
unique(
search(collection1, q=*:*, fl=id,a_s,a_i,a_f, sort=a_f desc),
over=a_f desc),
sort=a_f desc)
{code}
The syntax would support
1. Configurable expression names (eg. via solrconfig.xml one can map unique
to a class implementing a Unique stream class) This allows users to build
their own streams and use as they wish.
2. Named parameters (of both simple and expression types)
3. Unnamed, type-matched parameters (to support requiring N streams as
arguments to another stream)
4. Positional parameters
The main goal here is to make streaming as accessible as possible and define
a syntax for running complex queries across large distributed systems.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-7489) Don't wait as long to try and recover hdfs leases on transaction log files.
Mark Miller created SOLR-7489: - Summary: Don't wait as long to try and recover hdfs leases on transaction log files. Key: SOLR-7489 URL: https://issues.apache.org/jira/browse/SOLR-7489 Project: Solr Issue Type: Improvement Reporter: Mark Miller Assignee: Mark Miller Fix For: Trunk, 5.2 We initially just took most of this code from hbase which will wait for up to 15 minutes. This doesn't seem ideal - we should give up sooner and treat the file as not recoverable. We also need to fix the possible data loss message. This is really the same as if a transaction log on local disk were to become corrupt, and if you have a replica to recover from, things will be fine. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits
[ https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shalin Shekhar Mangar updated SOLR-6810: Attachment: SOLR-6810-trunk.patch Latest patch. The DistributedDebugComponentTest collects the filter_queries for every GET_FIELD requests. The existing DQA makes GET_FIELD requests only to those shards which contribute some documents to the results. However, the new DQA implementation makes a request to each shard regardless of whether it contributes a document or not. IMO, this is a bug (the extra request to every shard) but it is a bit tricky to solve for the following reasons: # A user can make a request with rows=0 in order to get just the facet responses. In such cases, we must make the STAGE_EXECUTE_QUERY/PURPOSE_GET_FIELDS requests to each shard. In fact many tests such as TestJsonFacets and some pivot facet tests do this. # Secondly, we must record things like numFound and maxScore from the results of the STAGE_LIMIT_ROWS itself if we are to skip requests to shards in the 2nd phase. At this stage, all tests pass and I am working on removing the nocommits (the biggest of which I detailed above). Faster searching limited but high rows across many shards all with many hits Key: SOLR-6810 URL: https://issues.apache.org/jira/browse/SOLR-6810 Project: Solr Issue Type: Improvement Components: search Reporter: Per Steffensen Assignee: Shalin Shekhar Mangar Labels: distributed_search, performance Attachments: SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, branch_5x_rev1645549.patch Searching limited but high rows across many shards all with many hits is slow E.g. * Query from outside client: q=somethingrows=1000 * Resulting in sub-requests to each shard something a-la this ** 1) q=somethingrows=1000fl=id,score ** 2) Request the full documents with ids in the global-top-1000 found among the top-1000 from each shard What does the subject mean * limited but high rows means 1000 in the example above * many shards means 200-1000 in our case * all with many hits means that each of the shards have a significant number of hits on the query The problem grows on all three factors above Doing such a query on our system takes between 5 min to 1 hour - depending on a lot of things. It ought to be much faster, so lets make it. Profiling show that the problem is that it takes lots of time to access the store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. Having 1000 shards its up to 1 mio ids that has to be fetched. There is really no good reason to ever read information from store for more than the overall top-1000 documents, that has to be returned to the client. For further detail see mail-thread Slow searching limited but high rows across many shards all with high hits started 13/11-2014 on dev@lucene.apache.org -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
[ https://issues.apache.org/jira/browse/SOLR-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519771#comment-14519771 ] David Smiley commented on SOLR-7488: [~hossman_luc...@fucit.org] I fixed this in SOLR-6692 (for v5.2) along with various other refactorings. It's now lazy initialized. This will save performance on wildcard queries too since FVH will expand it even if you will never use the FVH. suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used --- Key: SOLR-7488 URL: https://issues.apache.org/jira/browse/SOLR-7488 Project: Solr Issue Type: Bug Affects Versions: 4.10 Reporter: Hoss Man Rich Hume reported gettting errors from FastVectorHighlighter, evidently while using the the surround query parser, even though he was not trying to useFastVectorHighlighter my naive reading of the code leads me to believe that DefaultSolrHighlighter is incorrectly attempting to initialize a FVH instance even when it shouldn't be -- which appears to cause failures in cases where the query in use is not something that can be handled by the FVH. Not sure how to reproduce at the moment -- but the code smells fishy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519770#comment-14519770
]
Mark Miller commented on SOLR-7361:
---
Okay, with SolrCloud, if you hit a core that is loaded and one is not, you get:
{noformat}
?xml version=1.0 encoding=UTF-8?
response
lst name=responseHeaderint name=status503/intint
name=QTime3/intlst name=params//lstlst name=errorstr
name=msgno servers hosting shard: /strint name=code503/int/lst
/response
{noformat}
If you hit the core that is loading you get:
bq. curl: (7) Failed to connect to localhost port 8902: Connection refused
I'll work up a new patch.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Mark Miller
Fix For: Trunk, 5.2
Attachments: SOLR-7361.patch, SOLR-7361.patch, SOLR-7361.patch,
SOLR-7361.patch, SOLR-7361.patch
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Noble Paul updated SOLR-7484: - Attachment: SOLR-7484.patch more cleanup Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-5750) Backup/Restore API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker reassigned SOLR-5750: --- Assignee: Varun Thacker Backup/Restore API for SolrCloud Key: SOLR-5750 URL: https://issues.apache.org/jira/browse/SOLR-5750 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Shalin Shekhar Mangar Assignee: Varun Thacker Fix For: Trunk, 5.2 We should have an easy way to do backups and restores in SolrCloud. The ReplicationHandler supports a backup command which can create snapshots of the index but that is too little. The command should be able to backup: # Snapshots of all indexes or indexes from the leader or the shards # Config set # Cluster state # Cluster properties # Aliases # Overseer work queue? A restore should be able to completely restore the cloud i.e. no manual steps required other than bringing nodes back up or setting up a new cloud cluster. SOLR-5340 will be a part of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-5750) Backup/Restore API for SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-5750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Thacker updated SOLR-5750: Fix Version/s: (was: 4.9) 5.2 Backup/Restore API for SolrCloud Key: SOLR-5750 URL: https://issues.apache.org/jira/browse/SOLR-5750 Project: Solr Issue Type: Sub-task Components: SolrCloud Reporter: Shalin Shekhar Mangar Assignee: Varun Thacker Fix For: Trunk, 5.2 We should have an easy way to do backups and restores in SolrCloud. The ReplicationHandler supports a backup command which can create snapshots of the index but that is too little. The command should be able to backup: # Snapshots of all indexes or indexes from the leader or the shards # Config set # Cluster state # Cluster properties # Aliases # Overseer work queue? A restore should be able to completely restore the cloud i.e. no manual steps required other than bringing nodes back up or setting up a new cloud cluster. SOLR-5340 will be a part of this issue. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7458) Expose HDFS Block Locality Metrics
[
https://issues.apache.org/jira/browse/SOLR-7458?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mike Drob updated SOLR-7458:
Attachment: SOLR-7458.patch
Here's a version that caches the results for each file. We still have to do a
directory listing each time we get a JMX request, but that seems not too bad.
An alternative approach would be to update the cache only when we open a file
(i.e. {{new HdfsIndexInput()}}), but that had more intricate object life-cycle
implications so I decided to avoid it unless somebody else thought it might be
worthwhile.
Expose HDFS Block Locality Metrics
--
Key: SOLR-7458
URL: https://issues.apache.org/jira/browse/SOLR-7458
Project: Solr
Issue Type: Improvement
Components: SolrCloud
Reporter: Mike Drob
Assignee: Mark Miller
Labels: metrics
Attachments: SOLR-7458.patch, SOLR-7458.patch
We should publish block locality metrics when using HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7361) Main Jetty thread blocked by core loading delays HTTP listener from binding if core loading is slow
[
https://issues.apache.org/jira/browse/SOLR-7361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520221#comment-14520221
]
Mark Miller commented on SOLR-7361:
---
Made some good progress this afternoon. I'll post a patch relatively soon.
Main Jetty thread blocked by core loading delays HTTP listener from binding
if core loading is slow
---
Key: SOLR-7361
URL: https://issues.apache.org/jira/browse/SOLR-7361
Project: Solr
Issue Type: Bug
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Mark Miller
Fix For: Trunk, 5.2
Attachments: SOLR-7361.patch, SOLR-7361.patch, SOLR-7361.patch,
SOLR-7361.patch, SOLR-7361.patch
During server startup, the CoreContainer uses an ExecutorService to load
cores in multiple back-ground threads but then blocks until cores are loaded,
see: CoreContainer#load around line 290 on trunk (invokeAll). From the
JavaDoc on that method, we have:
{quote}
Executes the given tasks, returning a list of Futures holding their status
and results when all complete. Future.isDone() is true for each element of
the returned list.
{quote}
In other words, this is a blocking call.
This delays the Jetty HTTP listener from binding and accepting requests until
all cores are loaded. Do we need to block the main thread?
Also, prior to this happening, the node is registered as a live node in ZK,
which makes it a candidate for receiving requests from the Overseer, such as
to service a create collection request. The problem of course is that the
node listed in /live_nodes isn't accepting requests yet. So we either need to
unblock the main thread during server loading or maybe wait longer before we
register as a live node ... not sure which is the better way forward?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
Where Search Meets Machine Learning
Here is a presentation on the topic: http://www.slideshare.net/joaquindelgado1/where-search-meets-machine-learning04252015final Search can be viewed as a combination of a) A problem of constraint satisfaction, which is the process of finding a solution to a set of constraints (query) that impose conditions that the variables (fields) must satisfy with a resulting object (document) being a solution in the feasible region (result set), plus b) A scoring/ranking problem of assigning values to different alternatives, according to some convenient scale. This ultimately provides a mechanism to sort various alternatives in the result set in order of importance, value or preference. In particular scoring in search has evolved from being a document centric calculation (e.g. TF-IDF) proper from its information retrieval roots, to a function that is more context sensitive (e.g. include geo-distance ranking) or user centric (e.g. takes user parameters for personalization) as well as other factors that depend on the domain and task at hand. However, most system that incorporate machine learning techniques to perform classification or generate scores for these specialized tasks do so as a post retrieval re-ranking function, outside of search! In this talk I show ways of incorporating advanced scoring functions, based on supervised learning and bid scaling models, into popular search engines such as Elastic Search and potentially SOLR. I'll provide practical examples of how to construct such ML Scoring plugins in search to generalize the application of a search engine as a model evaluator for supervised learning tasks. This will facilitate the building of systems that can do computational advertising, recommendations and specialized search systems, applicable to many domains. Code to support it (only elastic search for now): https://github.com/sdhu/elasticsearch-prediction -- J
[JENKINS] Lucene-Solr-trunk-MacOSX (64bit/jdk1.8.0) - Build # 2241 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-MacOSX/2241/ Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.cloud.SyncSliceTest.test Error Message: expected:5 but was:4 Stack Trace: java.lang.AssertionError: expected:5 but was:4 at __randomizedtesting.SeedInfo.seed([DCC4C38CB3B2FF52:5490FC561D4E92AA]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.failNotEquals(Assert.java:647) at org.junit.Assert.assertEquals(Assert.java:128) at org.junit.Assert.assertEquals(Assert.java:472) at org.junit.Assert.assertEquals(Assert.java:456) at org.apache.solr.cloud.SyncSliceTest.test(SyncSliceTest.java:171) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:960) at org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:935) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at
[jira] [Updated] (SOLR-6968) add hyperloglog in statscomponent as an approximate count
[ https://issues.apache.org/jira/browse/SOLR-6968?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hoss Man updated SOLR-6968: --- Attachment: SOLR-6968.patch Updated patch with basic support for all field types (that are already supported by stats component) and some more tests. Also SolrJ response object getter method. add hyperloglog in statscomponent as an approximate count - Key: SOLR-6968 URL: https://issues.apache.org/jira/browse/SOLR-6968 Project: Solr Issue Type: Sub-task Reporter: Hoss Man Attachments: SOLR-6968.patch, SOLR-6968.patch stats component currently supports calcDistinct but it's terribly inefficient -- especially in distib mode. we should add support for using hyperloglog to compute an approximate count of distinct values (using localparams via SOLR-6349 to control the precision of the approximation) -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7349) Cleanup or fix cloud-dev scripts
[ https://issues.apache.org/jira/browse/SOLR-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519338#comment-14519338 ] ASF subversion and git services commented on SOLR-7349: --- Commit 1676742 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676742 ] SOLR-7349: svn ignore extra server folders Cleanup or fix cloud-dev scripts Key: SOLR-7349 URL: https://issues.apache.org/jira/browse/SOLR-7349 Project: Solr Issue Type: Improvement Components: scripts and tools Reporter: Ramkumar Aiyengar Assignee: Mark Miller Priority: Minor Fix For: 5.2 Attachments: SOLR-7349.patch, SOLR-7349.patch With Jetty 9, cloud-dev scripts no longer work in trunk, we need to either clean up or fix them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-Tests-5.x-Java7 - Build # 3031 - Failure
Build: https://builds.apache.org/job/Lucene-Solr-Tests-5.x-Java7/3031/ 1 tests failed. REGRESSION: org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds Error Message: soft529 wasn't fast enough Stack Trace: java.lang.AssertionError: soft529 wasn't fast enough at __randomizedtesting.SeedInfo.seed([AE7BEC915672CABE:FFAF1511E701FA19]:0) at org.junit.Assert.fail(Assert.java:93) at org.junit.Assert.assertTrue(Assert.java:43) at org.junit.Assert.assertNotNull(Assert.java:526) at org.apache.solr.update.SoftAutoCommitTest.testSoftAndHardCommitMaxTimeMixedAdds(SoftAutoCommitTest.java:111) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at java.lang.Thread.run(Thread.java:745) Build Log: [...truncated 9434 lines...] [junit4] Suite: org.apache.solr.update.SoftAutoCommitTest [junit4] 2 Creating dataDir:
[jira] [Commented] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[ https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519402#comment-14519402 ] David Smiley commented on LUCENE-6458: -- bq. Maybe we could change TermsQuery.rewrite to rewrite to a boolean query (wrapped in a CSQ) when there are few terms? This would avoid having to worry about this in every query parser. You read my mind ;-) I had that thought right after I posted. bq. In my opinion, the issue with rewriting to a BooleanQuery is that its scorer needs to rebalance the priority queue whenever it advances, which is O(log(#clauses)). So it gets slower as you add new optional clauses while the way TermsQuery works doesn't care much about the number of matching terms. I don't think the total number of index terms is relevant? Ah; I haven't looked in a while so I didn't know about the priority-queue over there in BooleanQuery (DisjunctionScorer actually). Never mind. MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6458.patch Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519349#comment-14519349 ] ASF subversion and git services commented on SOLR-7477: --- Commit 1676743 from [~yo...@apache.org] in branch 'dev/trunk' [ https://svn.apache.org/r1676743 ] SOLR-7477: implement facet excludeTags multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-6458) MultiTermQuery's FILTER rewrite method should support skipping whenever possible
[ https://issues.apache.org/jira/browse/LUCENE-6458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519358#comment-14519358 ] Adrien Grand commented on LUCENE-6458: -- I did some benchmarking and higher values (several hundreds) could make luceneutil run slower so I ended up with the same value that we are currently using for FuzzyQuery's max expansions. bq. This reminds me of when I was working on the Solr Terms QParser that supports 3-4 different options, to include BooleanQuery TermsQuery Maybe we could change TermsQuery.rewrite to rewrite to a boolean query (wrapped in a CSQ) when there are few terms? This would avoid having to worry about this in every query parser. bq. I have a feeling that the appropriate threshold is a function of the number of indexed terms, instead of just a constant. Hmm, what makes you think so? In my opinion, the issue with rewriting to a BooleanQuery is that its scorer needs to rebalance the priority queue whenever it advances, which is O(log(#clauses)). So it gets slower as you add new optional clauses while the way TermsQuery works doesn't care much about the number of matching terms. I don't think the total number of index terms is relevant? MultiTermQuery's FILTER rewrite method should support skipping whenever possible Key: LUCENE-6458 URL: https://issues.apache.org/jira/browse/LUCENE-6458 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Assignee: Adrien Grand Priority: Minor Attachments: LUCENE-6458.patch Today MultiTermQuery's FILTER rewrite always builds a bit set fom all matching terms. This means that we need to consume the entire postings lists of all matching terms. Instead we should try to execute like regular disjunctions when there are few terms. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[
https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519443#comment-14519443
]
Shalin Shekhar Mangar commented on SOLR-7477:
-
The following test seed always reproduces after your commit.
{code}
3616 T13 oas.SolrTestCaseHS.matchJSON ERROR query failed JSON validation.
error=mismatch: '3'!='2' @ facets/f2/count
expected =facets=={ 'count':2, 'f1':{'count':1, 'nj':{'count':1},
'ny':{'count':0}},'f2':{'count':3, 'nj':{'count':2},
'ny':{'count':1}},'f3':{'count':3, 'nj':{'count':2},
'ny':{'count':1}},'f4':{'count':3, 'nj':{'count':2},
'ny':{'count':1}},'f5':{'count':1, 'nj':{'count':1}, 'ny':{'count':0}}}
response = {
responseHeader:{
status:0,
QTime:13},
response:{numFound:2,start:0,maxScore:1.0,docs:[]
},
facets:{
count:2,
f1:{
count:1,
ny:{
count:0},
nj:{
count:1}},
f2:{
count:2,
ny:{
count:0},
nj:{
count:2}},
f3:{
count:2,
ny:{
count:0},
nj:{
count:2}},
f4:{
count:2,
ny:{
count:0},
nj:{
count:2}},
f5:{
count:1,
ny:{
count:0},
nj:{
count:1
3618 T13 oas.SolrTestCaseJ4.tearDown ###Ending testDistrib
java.lang.RuntimeException: mismatch: '3'!='2' @ facets/f2/count
at
__randomizedtesting.SeedInfo.seed([1C49DE2994FCEEBB:940A2DFCEF255D27]:0)
at org.apache.solr.SolrTestCaseHS.matchJSON(SolrTestCaseHS.java:160)
at org.apache.solr.SolrTestCaseHS.assertJQ(SolrTestCaseHS.java:142)
at org.apache.solr.SolrTestCaseHS$Client.testJQ(SolrTestCaseHS.java:288)
at
org.apache.solr.search.facet.TestJsonFacets.doStatsTemplated(TestJsonFacets.java:826)
at
org.apache.solr.search.facet.TestJsonFacets.doStats(TestJsonFacets.java:328)
at
org.apache.solr.search.facet.TestJsonFacets.testDistrib(TestJsonFacets.java:855)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627)
{code}
{code}
ant test -Dtestcase=TestJsonFacets -Dtests.method=testDistrib
-Dtests.seed=1C49DE2994FCEEBB -Dtests.locale=pt -Dtests.timezone=Asia/Hovd
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
{code}
multi-select support for facet module
-
Key: SOLR-7477
URL: https://issues.apache.org/jira/browse/SOLR-7477
Project: Solr
Issue Type: New Feature
Components: Facet Module
Reporter: Yonik Seeley
Fix For: 5.2
Attachments: SOLR-7477.patch
Multi-select support essentially means (at a minimum) support for excluding
tagged filters.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519454#comment-14519454 ] Yonik Seeley commented on SOLR-7477: Hmmm, that's strange. May be related to distrib search and not processing sub-facets for buckets w/ no count (of course if a sub-facet can expand the domain, then that makes things interesting). I'll fix when I can... but all my calls to svn are currently hanging. I assume it's an ASF prob and not on my end. multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Fwd: JEP 248: Make G1 the Default Garbage Collector
On 4/29/2015 12:45 AM, Dawid Weiss wrote: I think this may be of importance to us since we specifically don't like G1GC? :) I have been using G1 with Solr for quite some time and have had no trouble with it. I am aware of the note on the JavaBugs Lucene wiki page that says Do not, under any circumstances, run Lucene with the G1 garbage collector. There could be others that I haven't looked at, but the only Lucene bugs about G1GC I have noticed specifically mention the 32-bit implementation, which I am not using. I am very curious to know what kind of problems have been seen with G1 on a 64-bit JVM. Initially when I tried G1, it was completely untuned. It did not stop full GCs from happening, and the full GC pauses were longer than untuned CMS, though not as bad as leaving everything completely at default. I eventually learned how to tune CMS and ended up with really good performance from that. Much later, I figured out how to tune G1, and performance got even better. My notes on GC settings for Solr do specifically mention the Lucene recommendation to never use G1. http://wiki.apache.org/solr/ShawnHeisey#GC_Tuning_for_Solr Thanks, Shawn - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7332) Seed version buckets with max version from index
[
https://issues.apache.org/jira/browse/SOLR-7332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519457#comment-14519457
]
Timothy Potter commented on SOLR-7332:
--
Haven't been able to reproduce this with many stress tests on EC2 and it's
starting to get expensive ;-)
bq. Were there any recoveries or change of leaders during the run?
There definitely could have been some recoveries but I'm not sure. I'm taking a
snapshot of cluster state before I run my tests to compare to after in case I
do reproduce this. Yesterday I pushed it very hard with 48 reducers from
Hadoop, which led to some network issue between leader and replica and the
leader put the replica into recovery, see SOLR-7483. However, the replica
eventually recovered and was in-sync with the leader at the end, which is
goodness.
bq. No...
Thanks for confirming. I was thinking that maybe it had something to do with
this patch resetting the max after replaying the tlog:
From UpdateLog:
{code}
@@ -1247,6 +1269,12 @@
// change the state while updates are still blocked to prevent races
state = State.ACTIVE;
if (finishing) {
+
+ // after replay, update the max from the index
+ log.info(Re-computing max version from index after log re-play.);
+ maxVersionFromIndex = null;
+ getMaxVersionFromIndex();
+
versionInfo.unblockUpdates();
}
{code}
But since updates are blocked while this happens, it seems like the right thing
to do.
I'm going to run this a few more times using same setup as when it occurred the
first time and then I think we should commit this to trunk and see how it
behaves for a few days, as the performance improvement is a big win.
Seed version buckets with max version from index
Key: SOLR-7332
URL: https://issues.apache.org/jira/browse/SOLR-7332
Project: Solr
Issue Type: Sub-task
Components: SolrCloud
Reporter: Timothy Potter
Assignee: Timothy Potter
Attachments: SOLR-7332.patch, SOLR-7332.patch, SOLR-7332.patch,
SOLR-7332.patch, SOLR-7332.patch
See full discussion with Yonik and I in SOLR-6816.
The TL;DR of that discussion is that we should initialize highest for each
version bucket to the MAX value of the {{__version__}} field in the index as
early as possible, such as after the first soft- or hard- commit. This will
ensure that bulk adds where the docs don't exist avoid an unnecessary lookup
for a non-existent document in the index.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519467#comment-14519467 ] ASF subversion and git services commented on SOLR-7477: --- Commit 1676760 from [~yo...@apache.org] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676760 ] SOLR-7477: implement facet excludeTags multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519483#comment-14519483 ] ASF subversion and git services commented on SOLR-7477: --- Commit 1676763 from [~yo...@apache.org] in branch 'dev/trunk' [ https://svn.apache.org/r1676763 ] SOLR-7477: tests - processEmpty buckets to try and get distrib counts to match multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7477) multi-select support for facet module
[ https://issues.apache.org/jira/browse/SOLR-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519489#comment-14519489 ] ASF subversion and git services commented on SOLR-7477: --- Commit 1676764 from [~yo...@apache.org] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676764 ] SOLR-7477: tests - processEmpty buckets to try and get distrib counts to match multi-select support for facet module - Key: SOLR-7477 URL: https://issues.apache.org/jira/browse/SOLR-7477 Project: Solr Issue Type: New Feature Components: Facet Module Reporter: Yonik Seeley Fix For: 5.2 Attachments: SOLR-7477.patch Multi-select support essentially means (at a minimum) support for excluding tagged filters. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-7484: --- Attachment: SOLR-7484.patch Updated patch with a few changes. Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
[ https://issues.apache.org/jira/browse/SOLR-7486?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520735#comment-14520735 ] Shai Erera commented on SOLR-7486: -- Unless there are objections, I will commit this later today. HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 Attachments: SOLR-7486.patch HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anshum Gupta updated SOLR-7484: --- Attachment: SOLR-7484.patch Changed the access of most variables in SolrCall to be private and added a getter for path as it's accessed from the SDF. Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520681#comment-14520681 ] Anshum Gupta edited comment on SOLR-7484 at 4/30/15 1:36 AM: - I'm trying to see if there's a way to avoid re-processing during a RETRY but I think that can also come in later. was (Author: anshumg): I'm trying to see if there's a way to avoid processing during a RETRY but I think that can also come in later. Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7484) Refactor SolrDispatchFilter.doFilter(...) method
[ https://issues.apache.org/jira/browse/SOLR-7484?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520681#comment-14520681 ] Anshum Gupta commented on SOLR-7484: I'm trying to see if there's a way to avoid processing during a RETRY but I think that can also come in later. Refactor SolrDispatchFilter.doFilter(...) method Key: SOLR-7484 URL: https://issues.apache.org/jira/browse/SOLR-7484 Project: Solr Issue Type: Improvement Reporter: Anshum Gupta Assignee: Anshum Gupta Attachments: SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch, SOLR-7484.patch Currently almost everything that's done in SDF.doFilter() is sequential. We should refactor it to clean up the code and make things easier to manage. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Windows (32bit/jdk1.8.0_45) - Build # 4744 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/4744/ Java: 32bit/jdk1.8.0_45 -server -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.search.facet.TestJsonFacets.testDistrib Error Message: mismatch: '3'!='1' @ facets/f2/count Stack Trace: java.lang.RuntimeException: mismatch: '3'!='1' @ facets/f2/count at __randomizedtesting.SeedInfo.seed([136637CB84E42CE7:9B25C41EFF3D9F7B]:0) at org.apache.solr.SolrTestCaseHS.matchJSON(SolrTestCaseHS.java:160) at org.apache.solr.SolrTestCaseHS.assertJQ(SolrTestCaseHS.java:142) at org.apache.solr.SolrTestCaseHS$Client.testJQ(SolrTestCaseHS.java:288) at org.apache.solr.search.facet.TestJsonFacets.doStatsTemplated(TestJsonFacets.java:826) at org.apache.solr.search.facet.TestJsonFacets.doStats(TestJsonFacets.java:328) at org.apache.solr.search.facet.TestJsonFacets.testDistrib(TestJsonFacets.java:855) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at
[jira] [Created] (SOLR-7486) HttpSolrClient.shutdown() should call close() and not vice versa
Shai Erera created SOLR-7486: Summary: HttpSolrClient.shutdown() should call close() and not vice versa Key: SOLR-7486 URL: https://issues.apache.org/jira/browse/SOLR-7486 Project: Solr Issue Type: Bug Reporter: Shai Erera Assignee: Shai Erera Fix For: 5.2 HttpSolrClient.shutdown() is deprecated, however close() calls it instead of the other way around. If anyone extends HttpSolrClient, he needs to override both methods to make sure things are closed properly. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7488) suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used
[ https://issues.apache.org/jira/browse/SOLR-7488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520776#comment-14520776 ] Koji Sekiguchi commented on SOLR-7488: -- Thanks David and Hoss! suspicious FVH init code in DefaultSolrHighlighter even when FVH should not be used --- Key: SOLR-7488 URL: https://issues.apache.org/jira/browse/SOLR-7488 Project: Solr Issue Type: Bug Affects Versions: 4.10 Reporter: Hoss Man Assignee: David Smiley Fix For: Trunk, 5.2 Rich Hume reported gettting errors from FastVectorHighlighter, evidently while using the the surround query parser, even though he was not trying to useFastVectorHighlighter my naive reading of the code leads me to believe that DefaultSolrHighlighter is incorrectly attempting to initialize a FVH instance even when it shouldn't be -- which appears to cause failures in cases where the query in use is not something that can be handled by the FVH. Not sure how to reproduce at the moment -- but the code smells fishy. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7490) Update by query feature
[
https://issues.apache.org/jira/browse/SOLR-7490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520812#comment-14520812
]
Erick Erickson commented on SOLR-7490:
--
Problem here is that you don't necessarily have any source to re-index from.
Solr doesn't (yet) have any update-in-place, atomic updates only work if all
source fields are stored. So this could only possibly work if all the fields in
the docs were stored, and could well result in a _massive_ amount of work being
done by Solr as the result of a single call.
I guess it might be possible to accommodate this with the updatable docValues,
but again that would be a restricted use-case.
It feels like the restrictions are too onerous to make this something that
would generate a lot of interest. I'm not opposed to the idea, mind you, but I
_do_ need to be persuaded that it's worth the complexity.
And how that would all play with distributed updates and optimistic locking I
don't know.
Update by query feature
---
Key: SOLR-7490
URL: https://issues.apache.org/jira/browse/SOLR-7490
Project: Solr
Issue Type: New Feature
Reporter: Praneeth
Priority: Minor
An update feature similar to the {{deleteByQuery}} would be very useful. Say,
the user wants to update a field of all documents in the index that match a
given criteria. I have encountered this use case in my project and it looks
like it could be a useful first class solr/lucene feature. I want to check if
this is something we would want to support in coming releases of Solr and
Lucene, are there scenarios that will prevent us from doing this,
feasibility, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[JENKINS] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b60) - Build # 12501 - Failure!
Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/12501/ Java: 32bit/jdk1.9.0-ea-b60 -server -XX:+UseG1GC 1 tests failed. FAILED: org.apache.solr.search.facet.TestJsonFacets.testDistrib Error Message: mismatch: '3'!='2' @ facets/f2/count Stack Trace: java.lang.RuntimeException: mismatch: '3'!='2' @ facets/f2/count at __randomizedtesting.SeedInfo.seed([8B6FA9C75A02099E:32C5A1221DBBA02]:0) at org.apache.solr.SolrTestCaseHS.matchJSON(SolrTestCaseHS.java:160) at org.apache.solr.SolrTestCaseHS.assertJQ(SolrTestCaseHS.java:142) at org.apache.solr.SolrTestCaseHS$Client.testJQ(SolrTestCaseHS.java:288) at org.apache.solr.search.facet.TestJsonFacets.doStatsTemplated(TestJsonFacets.java:826) at org.apache.solr.search.facet.TestJsonFacets.doStats(TestJsonFacets.java:328) at org.apache.solr.search.facet.TestJsonFacets.testDistrib(TestJsonFacets.java:855) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:502) at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1627) at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:836) at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:872) at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:886) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:798) at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:458) at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:845) at com.carrotsearch.randomizedtesting.RandomizedRunner$3.evaluate(RandomizedRunner.java:747) at com.carrotsearch.randomizedtesting.RandomizedRunner$4.evaluate(RandomizedRunner.java:781) at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:792) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57) at org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:39) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54) at org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48) at org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65) at org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55) at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36) at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:365) at
[jira] [Commented] (SOLR-7349) Cleanup or fix cloud-dev scripts
[ https://issues.apache.org/jira/browse/SOLR-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519502#comment-14519502 ] ASF subversion and git services commented on SOLR-7349: --- Commit 1676765 from [~markrmil...@gmail.com] in branch 'dev/trunk' [ https://svn.apache.org/r1676765 ] SOLR-7349: Additional updates. Cleanup or fix cloud-dev scripts Key: SOLR-7349 URL: https://issues.apache.org/jira/browse/SOLR-7349 Project: Solr Issue Type: Improvement Components: scripts and tools Reporter: Ramkumar Aiyengar Assignee: Mark Miller Priority: Minor Fix For: 5.2 Attachments: SOLR-7349.patch, SOLR-7349.patch With Jetty 9, cloud-dev scripts no longer work in trunk, we need to either clean up or fix them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7349) Cleanup or fix cloud-dev scripts
[ https://issues.apache.org/jira/browse/SOLR-7349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519505#comment-14519505 ] ASF subversion and git services commented on SOLR-7349: --- Commit 1676766 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x' [ https://svn.apache.org/r1676766 ] SOLR-7349: Additional updates. Cleanup or fix cloud-dev scripts Key: SOLR-7349 URL: https://issues.apache.org/jira/browse/SOLR-7349 Project: Solr Issue Type: Improvement Components: scripts and tools Reporter: Ramkumar Aiyengar Assignee: Mark Miller Priority: Minor Fix For: 5.2 Attachments: SOLR-7349.patch, SOLR-7349.patch With Jetty 9, cloud-dev scripts no longer work in trunk, we need to either clean up or fix them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-3956) Urgent Lucene Exception Issue
[ https://issues.apache.org/jira/browse/LUCENE-3956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Erick Erickson resolved LUCENE-3956. Resolution: Not A Problem Happened across this and it doesn't seem useful to keep open. Urgent Lucene Exception Issue - Key: LUCENE-3956 URL: https://issues.apache.org/jira/browse/LUCENE-3956 Project: Lucene - Core Issue Type: Bug Reporter: Dylan SD Hello, Could you please advise how this issue: TP-Processor13 ERROR [com.atlassian.bonnie.ConcurrentLuceneConnection] Error closing writer. java.io.FileNotFoundException: /opt/jira/indexes/comments/_4ubij.fnm (No such file or directory) java.io.FileNotFoundException: /opt/jira/indexes/comments/_4ubij.fnm (No such file or directory) Thread-228 ERROR [atlassian.core.user.UserUtils] [Ljava.lang.StackTraceElement;@258e5d0e Thread-228 ERROR [event.listeners.search.IssueIndexListener] Error re-indexing changes for issue 'current issue' com.atlassian.jira.issue.index.IndexException: com.atlassian.bonnie.LuceneException: org.apache.lucene.store.LockObtainFailedException: Lock obtain timed out: SimpleFSLock@/opt/jira/indexes/comments/write.lock can be resolved in JIRA. I've deleted the write.lock file from directory /opt/jira/indexes/comments/ and restarted JIRA but it did not help. The problem with Lucene still exists. Please advise what caused the problem and how it can be avoid in the future and what is more important how this Lucene Exception can be resolved. Regards, Dylan -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-7275) Pluggable authorization module in Solr
[
https://issues.apache.org/jira/browse/SOLR-7275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14519516#comment-14519516
]
Ishan Chattopadhyaya commented on SOLR-7275:
{quote}
{noformat}
+ if (!authResponse.isAuthorized()) {
+sendError((HttpServletResponse) response, 401, Unauthorized request);
{noformat}
{quote}
Usually, a 401 means that the request needs to be repeated with appropriate
authentication/authorization headers. 403 means that the
authentication/authorization headers that are needed were provided, but still
the server refused to fulfill the request.
I think 403 (Forbidden) is more appropriate here.
Pluggable authorization module in Solr
--
Key: SOLR-7275
URL: https://issues.apache.org/jira/browse/SOLR-7275
Project: Solr
Issue Type: Sub-task
Reporter: Anshum Gupta
Assignee: Anshum Gupta
Attachments: SOLR-7275.patch, SOLR-7275.patch, SOLR-7275.patch
Solr needs an interface that makes it easy for different authorization
systems to be plugged into it. Here's what I plan on doing:
Define an interface {{SolrAuthorizationPlugin}} with one single method
{{isAuthorized}}. This would take in a {{SolrRequestContext}} object and
return an {{SolrAuthorizationResponse}} object. The object as of now would
only contain a single boolean value but in the future could contain more
information e.g. ACL for document filtering etc.
The reason why we need a context object is so that the plugin doesn't need to
understand Solr's capabilities e.g. how to extract the name of the collection
or other information from the incoming request as there are multiple ways to
specify the target collection for a request. Similarly request type can be
specified by {{qt}} or {{/handler_name}}.
Flow:
Request - SolrDispatchFilter - isAuthorized(context) - Process/Return.
{code}
public interface SolrAuthorizationPlugin {
public SolrAuthorizationResponse isAuthorized(SolrRequestContext context);
}
{code}
{code}
public class SolrRequestContext {
UserInfo; // Will contain user context from the authentication layer.
HTTPRequest request;
Enum OperationType; // Correlated with user roles.
String[] CollectionsAccessed;
String[] FieldsAccessed;
String Resource;
}
{code}
{code}
public class SolrAuthorizationResponse {
boolean authorized;
public boolean isAuthorized();
}
{code}
User Roles:
* Admin
* Collection Level:
* Query
* Update
* Admin
Using this framework, an implementation could be written for specific
security systems e.g. Apache Ranger or Sentry. It would keep all the security
system specific code out of Solr.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
