[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_66) - Build # 5391 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/5391/
Java: 64bit/jdk1.8.0_66 -XX:-UseCompressedOops -XX:+UseG1GC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
ERROR: SolrIndexSearcher opens=51 closes=50

Stack Trace:
java.lang.AssertionError: ERROR: SolrIndexSearcher opens=51 closes=50
at __randomizedtesting.SeedInfo.seed([14A9DFDD0DFB5022]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.SolrTestCaseJ4.endTrackingSearchers(SolrTestCaseJ4.java:468)
at org.apache.solr.SolrTestCaseJ4.afterClass(SolrTestCaseJ4.java:234)
at sun.reflect.GeneratedMethodAccessor21.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
1 thread leaked from SUITE scope at org.apache.solr.core.TestLazyCores: 1) 
Thread[id=13326, name=searcherExecutor-5941-thread-1, state=WAITING, 
group=TGRP-TestLazyCores] at sun.misc.Unsafe.park(Native Method)
 at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.core.TestLazyCores: 
   1) Thread[id=13326, name=searcherExecutor-5941-thread-1, state=WAITING, 
group=TGRP-TestLazyCores]
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([14A9DFDD0DFB5022]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.core.TestLazyCores

Error Message:
There are still zombie threads that couldn't be terminated:1) 
Thread[id=13326, 

[jira] [Updated] (LUCENE-6891) Lucene30DimensionalFormat should use block prefix coding when writing values

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-6891:
---
Attachment: LUCENE-6891.patch

Patch, tests/precommit passes, I think it's ready.

> Lucene30DimensionalFormat should use block prefix coding when writing values
> 
>
> Key: LUCENE-6891
> URL: https://issues.apache.org/jira/browse/LUCENE-6891
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
> Fix For: Trunk
>
> Attachments: LUCENE-6891.patch
>
>
> Today we write the whole value for every doc in one leaf block in the BKD 
> tree, but that's crazy because the whole point of that leaf block is all the 
> docs inside it have values that are very close together.
> So I changed this to write the common prefix for the whole block up front in 
> each block.  This requires more index-time and search-time work, but gives 
> nice index size reductions:
> On the 2D (London, UK) lat/lon benchmark:
>   * Indexing time was a wee bit slower (743 -> 747 seconds)
>   * Index size was ~11% smaller (704 MB -> 630 MB)
>   * Query time was ~7% slower (2.84 sec -> 3.05 sec)
>   * Heap usage is the same
> On the 1D (just "lat" from the above test) benchmark:
>   * Indexing time was a wee bit slower (363 -> 364 sec)
>   * Index size was ~23% smaller (472 MB -> 363 MB)
>   * Query time was a wee bit slower (5.39 -> 5.41 sec)
>   * Heap usage is the same
> Index time can be a bit slower since there are two passes now per leaf block 
> (first to find the common prefix per dimension, and second pass must then 
> strip those prefixes).
> Query time is slower because there's more work per hit that needs value 
> filtering, i.e. collating the suffixes onto the prefixes, per dimension.  
> This affects 2D much more than 1D because 1D has fewer leaf blocks that need 
> filtering (typically 0, 1 or 2, unless there are many duplicate values in the 
> index).
> I suspect the index size savings is use-case dependent, e.g. if you index a 
> bunch of ipv4 addresses along with a few ipv6 addresses, you'd probably see 
> sizable savings.
> I also suspect the more docs you index, the greater the savings, because the 
> cells will generally be smaller.
> Net/net I think the opto is worth it, even if it slows query time a bit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  \{!djoin\}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  \{!djoin\}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the {!djoin} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler 

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000256#comment-15000256
 ] 

Noble Paul commented on SOLR-7989:
--

99/100 times the state=ACTIVE

you can do a force read before checking the state .

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the {{/select}} 
request handler, we use the {{FilterQParserSearchComponent}} which is 
configured to filter out the {{{!djoin}}} search parameter - otherwise, the 
target request handler complains about the missing query parser definition. See 
the example config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]
 

[jira] [Updated] (SOLR-8235) Federated Search (new) - Merge

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8235:

Flags:   (was: Patch)

> Federated Search (new) - Merge
> --
>
> Key: SOLR-8235
> URL: https://issues.apache.org/jira/browse/SOLR-8235
> Project: Solr
>  Issue Type: New Feature
>Reporter: Tom Winch
>Priority: Minor
>  Labels: federated_search
> Fix For: 4.10.3
>
>
> This issue describes a SearchComponent for merging search results obtained 
> from a DJoin distributed search (see SOLR-8234) as part of federated search - 
> that is, distributed search over documents stored in separated instances of 
> SOLR (for example, one server per continent), where a single document 
> (identified by an agreed, common unique id) may be stored in more than one 
> server instance, with (possibly) differing fields and data.
> In the use of this search component, it is assumed that there is a single 
> SOLR server (the "aggregator") that uses distributed search (shards=) to 
> collect documents from other SOLR servers using DJoin (see SOLR-8234). The 
> DJoin generates a result set containing parent documents each with child 
> documents having the same unique id. This merge component turns each set of 
> child documents into a single document conforming to the aggregator schema.
> For example, suppose the aggregator schema defines a multi-valued integer 
> field, 'num', and three shards return field values "48", 23, and 
> "strawberry". Then the resulting merged field value would be [48, 23] and an 
> error would included for the NumberFormatException.
> Custom field merge behaviour may be specified by defining custom field types 
> in the usual way and this is supported via a MergeAbstractFieldType class.
> This issue combines with others to provide full federated search support. See 
> also SOLR-8234 and SOLR-8236.
> –
> Note that this is part of a new implementation of federated search as opposed 
> to the older issues SOLR-3799 through SOLR-3805.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{{
  
djoin
  
  
  

  


  shard/1,shard/2,shard/3
  false
  {!djoin}


  filter

   
}}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

--

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.


> Federated Search (new) - DJoin
> --
>
> Key: SOLR-8234
> URL: https://issues.apache.org/jira/browse/SOLR-8234
> Project: Solr
>  Issue Type: New Feature
>Reporter: Tom Winch
>Priority: Minor
>  Labels: federated_search
> Fix For: 4.10.3
>
> Attachments: SOLR-8234.patch
>
>
> This issue describes a MergeStrategy implementation (DJoin) to facilitate 
> federated search - that is, distributed search over documents stored in 
> separated instances of SOLR (for example, one server per continent), where a 
> single document (identified by an agreed, common unique id) may be stored in 
> more than one server instance, with (possibly) differing fields and data.
> When the MergeStrategy is used in a request handler (via the included 
> QParser) in combination with distributed search (shards=), documents having 
> an id that has already been seen are not discarded (as per the default 
> behaviour) but, instead, are collected and returned as a group of documents 
> all with the same id taking a single position in the result set (this is 
> implemented using parent/child documents).
> Documents are sorted in the result set based on the highest ranking document 
> with the same id. It is possible for a document ranking high in one shard to 
> rank very low on another shard. As a consequence of this, all shards must be 
> asked to return the fields for every document id in the result set (not just 
> of those documents they returned), so that all the component parts of each 
> document in the search result set are 

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the {!djoin} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the {{/select}} 
request handler, we use the {{FilterQParserSearchComponent}} which is 
configured to filter out the {{{!djoin}}} search parameter - otherwise, the 
target request handler complains about the missing query parser definition. See 
the example config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request 

[JENKINS] Lucene-Solr-NightlyTests-trunk - Build # 849 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-trunk/849/

1 tests failed.
FAILED:  org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([3C8DF2D38AD7E76:EA926415A634EEDE]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:766)
at 
org.apache.solr.core.TestArbitraryIndexDir.testLoadNewIndexDir(TestArbitraryIndexDir.java:128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: xpath=*[count(//doc)=1]
xml response was: 

00


request was:q=id:2=standard=0=20=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:759)
... 40 more




Build Log:
[...truncated 10899 lines...]
   [junit4] Suite: org.apache.solr.core.TestArbitraryIndexDir
   [junit4]   2> Creating 

Re: Inconsistent error returned by Solr when indexing bad documents

[Mikhail Khludnev]

Hello,

Here is my attempt. It might be too heavy but it proves the problem at
least.
https://paste.apache.org/dnHa


On Wed, Nov 11, 2015 at 8:59 AM, Shai Erera  wrote:

> OK just wanted to clarify that this happens even when indexing a single
> document, using curl or SolrJ:
>
> curl -i -X POST http://localhost:7574/solr/gettingstarted/update/json/docs
> -d '{"id" : "1", "unknown" : "foo"}'
>
> So it's not only with bulk updates.
>
> I will nevertheless reproduce this in a proper unit test.
>
> Shai
>
> On Tue, Nov 10, 2015 at 6:31 PM, Mark Miller 
> wrote:
>
>> It's kind of tricky. And it doesn't help that we don't have real per
>> update errors when doing bulk or streaming.
>>
>> But you can start looking around DistributedUpdateProcessor,
>> SolrCmdDistributor, StreamingSolrClients for update error propagation
>> 'stuff'.
>>
>> - Mark
>>
>> On Tue, Nov 10, 2015 at 10:39 AM Shai Erera  wrote:
>>
>>> Thanks Mark, I wrote a test for it, I can port it to Solr's
>>> test-framework. Can you also give me a hint in what area of the code I
>>> should look to fix it?
>>>
>>> Shai
>>>
>>> On Tue, Nov 10, 2015 at 4:59 PM, Mark Miller 
>>> wrote:
>>>
 It's not properly propagating the root error it loos. We probably need
 a test for that.

 - Mark

 On Tue, Nov 10, 2015 at 7:54 AM Shai Erera  wrote:

> Hi,
>
> I wanted to test the error message that Solr returns when indexing a
> document with an unknown field. Surprisingly, I get different errors,
> depending if the request hits the shard's leader or not.
>
> To reproduce (5.3.1):
>
> bin/solr -e cloud
>   ports: 8983, 7574
>   config: basic_configs
>   shards: 1
>   replicas: 2
>
> Wait for the nodes to come up and issue a CLUSTERSTATUS call to check
> which replica is the leader. In my case, 7574 was the leader. Now
> index a document with an unknown field:
>
> curl -i -X POST http://localhost:8983/solr/gettingstarted/update/json
> -d '[{"id" : "1", "unknown" : "foo"}]'
>
> And you get back
>
> {"responseHeader":{"status":400,"QTime":6},"error":{"msg":"Bad
> Request\n\n\n\nrequest:
> http://169.254.21.228:7574/solr/gettingstarted_shard1_replica1/update?update.distrib=TOLEADER=http%3A%2F%2F169.254.21.228%3A8983%2Fsolr%2Fgettingstarted_shard1_replica2%2F=javabin=2
> ","code":400}}
>
> But if you execute:
>
> curl -i -X POST http://localhost:7574/solr/gettingstarted/update/json
> -d '[{"id" : "1", "unknown" : "foo"}]'
>
> Then you get back
>
> {"responseHeader":{"status":400,"QTime":1},"error":{"msg":"ERROR:
> [doc=1] unknown field 'unknown'","code":400}}
>
> In both cases you get back 400, but if the request hits the leader you
> get a more expressive error message. Is there any reason for that 
> behavior?
> Can't the replica just pass along the error that it got from the leader?
>
>
> Shai
>
 --
 - Mark
 about.me/markrmiller

>>>
>>> --
>> - Mark
>> about.me/markrmiller
>>
>
>


-- 
Sincerely yours
Mikhail Khludnev
Principal Engineer,
Grid Dynamics

[JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 180 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/180/
Java: multiarch/jdk1.7.0 -d64 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior

Error Message:


Stack Trace:
java.lang.NullPointerException
at 
__randomizedtesting.SeedInfo.seed([B8F1A293A26DD72E:D0EFA17F40FD8D60]:0)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.verifyReplicaStatus(AbstractDistribZkTestBase.java:222)
at 
org.apache.solr.cloud.OverseerTest.testExternalClusterStateChangeBehavior(OverseerTest.java:1244)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)




Build Log:
[...truncated 10570 lines...]
   [junit4] Suite: org.apache.solr.cloud.OverseerTest
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.OverseerTest_B8F1A293A26DD72E-001/init-core-data-001
   [junit4]   2> 

[jira] [Updated] (SOLR-7341) xjoin - join data from external sources

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7341?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-7341:

Flags: Patch

> xjoin - join data from external sources
> ---
>
> Key: SOLR-7341
> URL: https://issues.apache.org/jira/browse/SOLR-7341
> Project: Solr
>  Issue Type: New Feature
>  Components: search
>Affects Versions: 4.10.3
>Reporter: Tom Winch
>Priority: Minor
> Fix For: Trunk
>
> Attachments: SOLR-7341.patch, SOLR-7341.patch, SOLR-7341.patch, 
> SOLR-7341.patch, SOLR-7341.patch, SOLR-7341.patch, SOLR-7341.patch-trunk, 
> SOLR-7341.patch-trunk, SOLR-7341.patch-trunk
>
>
> h2. XJoin
> The "xjoin" SOLR contrib allows external results to be joined with SOLR 
> results in a query and the SOLR result set to be filtered by the results of 
> an external query. Values from the external results are made available in the 
> SOLR results and may also be used to boost the scores of corresponding 
> documents during the search. The contrib consists of the Java classes 
> XJoinSearchComponent, XJoinValueSourceParser and XJoinQParserPlugin (and 
> associated classes), which must be configured in solrconfig.xml, and the 
> interfaces XJoinResultsFactory and XJoinResults, which are implemented by the 
> user to provide the link between SOLR and the external results source. 
> External results and SOLR documents are matched via a single configurable 
> attribute (the "join field"). The contrib JAR solr-xjoin-4.10.3.jar contains 
> these classes and interfaces and should be included in SOLR's class path from 
> solrconfig.xml, as should a JAR containing the user implementations of the 
> previously mentioned interfaces. For example:
> {code:xml}
> 
>   ..
>   
>/>
>   ..
>   
>   
>   ..
> 
> {code}
> h2. Java classes and interfaces
> h3. XJoinResultsFactory
> The user implementation of this interface is responsible for connecting to an 
> external source to perform a query (or otherwise collect results). Parameters 
> with prefix ".external." are passed from the SOLR query URL 
> to pararameterise the search. The interface has the following methods:
> * void init(NamedList args) - this is called during SOLR initialisation, and 
> passed parameters from the search component configuration (see below)
> * XJoinResults getResults(SolrParams params) - this is called during a SOLR 
> search to generate external results, and is passed parameters from the SOLR 
> query URL (as above)
> For example, the implementation might perform queries of an external source 
> based on the 'q' SOLR query URL parameter (in full,  name>.external.q).
> h3. XJoinResults
> A user implementation of this interface is returned by the getResults() 
> method of the XJoinResultsFactory implementation. It has methods:
> * Object getResult(String joinId) - this should return a particular result 
> given the value of the join attribute
> * Iterable getJoinIds() - this should return an ordered (ascending) 
> list of the join attribute values for all results of the external search
> h3. XJoinSearchComponent
> This is the central Java class of the contrib. It is a SOLR search component, 
> configured in solrconfig.xml and included in one or more SOLR request 
> handlers. There is one XJoin search component per external source, and each 
> has two main responsibilities:
> * Before the SOLR search, it connects to the external source and retrieves 
> results, storing them in the SOLR request context
> * After the SOLR search, it matches SOLR document in the results set and 
> external results via the join field, adding attributes from the external 
> results to documents in the SOLR results set
> It takes the following initialisation parameters:
> * factoryClass - this specifies the user-supplied class implementing 
> XJoinResultsFactory, used to generate external results
> * joinField - this specifies the attribute on which to join between SOLR 
> documents and external results
> * external - this parameter set is passed to configure the 
> XJoinResultsFactory implementation
> For example, in solrconfig.xml:
> {code:xml}
>  class="org.apache.solr.search.xjoin.XJoinSearchComponent">
>   test.TestXJoinResultsFactory
>   id
>   
> 1,2,3
>   
> 
> {code}
> Here, the search component instantiates a new TextXJoinResultsFactory during 
> initialisation, and passes it the "values" parameter (1, 2, 3) to configure 
> it. To properly use the XJoinSearchComponent in a request handler, it must be 
> included at the start and end of the component list, and may be configured 
> with the following query parameters:
> * results - a comma-separated list of attributes from the XJoinResults 
> implementation (created by the factory at search time) to be included in the 
> SOLR results
> * fl - a comma-separated list of attributes from results 

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  



  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{{
  
djoin
  
  
  

  


  shard/1,shard/2,shard/3
  false
  {!djoin}


  filter

   
}}


> Federated Search (new) - DJoin
> --
>
> Key: SOLR-8234
> URL: https://issues.apache.org/jira/browse/SOLR-8234
> Project: Solr
>  Issue Type: New Feature
>Reporter: Tom Winch
>Priority: Minor
>  Labels: federated_search
> Fix For: 4.10.3
>
> Attachments: SOLR-8234.patch
>
>
> This issue describes a MergeStrategy implementation (DJoin) to facilitate 
> federated search - that is, distributed search over documents stored in 
> separated instances of SOLR (for example, one server per continent), where a 
> single document 

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler 

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  \{!djoin\}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler 

[jira] [Updated] (LUCENE-6892) various lucene.index initialCapacity tweaks

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated LUCENE-6892:

Attachment: LUCENE-6892.patch

proposed patch against trunk

> various lucene.index initialCapacity tweaks
> ---
>
> Key: LUCENE-6892
> URL: https://issues.apache.org/jira/browse/LUCENE-6892
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6892.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 179 - Still Failing!

[Dawid Weiss]

Sigh. Thanks Jan, this wasn't my top performance day, was it...

Dawid

On Wed, Nov 11, 2015 at 12:08 AM, Jan Høydahl  wrote:
> I committed fixes for this (SerbianFilter), so should pass next time...
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
>> 10. nov. 2015 kl. 23.59 skrev Policeman Jenkins Server :
>>
>> Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/179/
>> Java: multiarch/jdk1.7.0 -d32 -server -XX:+UseParallelGC
>>
>> All tests passed
>>
>> Build Log:
>> [...truncated 20280 lines...]
>> BUILD FAILED
>> /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:785: The 
>> following error occurred while executing this line:
>> /export/home/jenkins/workspace/Lucene-Solr-5.x-Solaris/build.xml:130: Found 
>> 1 violations in source files (tabs instead spaces).
>>
>> Total time: 91 minutes 52 seconds
>> Build step 'Invoke Ant' marked build as failure
>> Archiving artifacts
>> [WARNINGS] Skipping publisher since build result is FAILURE
>> Recording test results
>> Email was triggered for: Failure - Any
>> Sending email for trigger: Failure - Any
>>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6892) various lucene.index initialCapacity tweaks

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000349#comment-15000349
 ] 

Uwe Schindler commented on LUCENE-6892:
---

+1 Schaut gut aus!

> various lucene.index initialCapacity tweaks
> ---
>
> Key: LUCENE-6892
> URL: https://issues.apache.org/jira/browse/LUCENE-6892
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6892.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reopened SOLR-8075:
---

There appears to be some race or something here. We need to turn off LIR 
*before* we try and publish ACTIVE upon becoming the leader. The test must only 
pass because somehow the check beats the ACTIVE publish or something.

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8225) Leader should send update requests to replicas in recovery asynchronously

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000659#comment-15000659
 ] 

Mark Miller commented on SOLR-8225:
---

bq. but the leader had to wait for the slowest replica.

To some degree, you could mitigate this with the min replication factor on 
updates right? I don't know how it is now, but in an ideal world, we could 
return once we got the n fastest responses.

> Leader should send update requests to replicas in recovery asynchronously
> -
>
> Key: SOLR-8225
> URL: https://issues.apache.org/jira/browse/SOLR-8225
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Reporter: Timothy Potter
>
> When a replica goes into recovery, the leader still sends docs to that 
> replica while it is recovering. What I'm seeing is that the recovering node 
> is still slow to respond to the leader (at least slower than the healthy 
> replicas). Thus it would be good if the leader could send the updates to the 
> recovering replica asynchronously, i.e. the leader will block as it does 
> today when forwarding updates to healthy / active replicas, but send updates 
> to recovering replicas async, thus preventing the whole update request from 
> being slowed down by a potentially degraded.
> FWIW - I've actually seen this occur in an environment that has more than 3 
> replicas per shard. One of the replicas went into recovery and then was much 
> slower to handle requests than the healthy replicas, but the leader had to 
> wait for the slowest replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7951) LBHttpSolrClient wraps ALL exceptions in "No live SolrServers available to handle this request" exception, even usage errors

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000673#comment-15000673
 ] 

Mark Miller commented on SOLR-7951:
---

Sorry guys - just have not had a chance to dig back into this one. If someone 
else doesn't pick it up, it's still on my list.

> LBHttpSolrClient wraps ALL exceptions in "No live SolrServers available to 
> handle this request" exception, even usage errors
> 
>
> Key: SOLR-7951
> URL: https://issues.apache.org/jira/browse/SOLR-7951
> Project: Solr
>  Issue Type: Bug
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Elaine Cario
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-7951-4.x.patch, SOLR-7951-4.x.patch, SOLR-7951.patch
>
>
> We were experiencing many "No live SolrServers available to handle this 
> request" exception, even though we saw no outages with any of our servers.  
> It turned out the actual exceptions were related to the use of wildcards in 
> span queries (and in some cases other invalid queries or usage-type issues). 
> Traced it back to LBHttpSolrClient which was wrapping all exceptions, even 
> plain SolrExceptions, in that outer exception.  
> Instead, wrapping in the out exception should be reserved for true 
> communication issues in SolrCloud, and usage exceptions should be thrown as 
> is.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000683#comment-15000683
 ] 

Mark Miller commented on SOLR-8075:
---

The latest patch also fixes LIR handling a bit. The way things worked were a 
little off.

If you were the first replica to attempt to be leader, LIR would never stop you 
early. Instead, things would fail when publish ACTIVE was attempted. That is 
not good, because it will probably prevent the next leader from trying to take 
over.

So I have moved all the LIR checking code in ElectionContext to a new spot. 
It's always checked, first attempted leader or not. And not until we see if the 
replica *could* be leader and if all the replicas participated in that election 
or not.

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-6893) replace CorePlusExtensionsParser with CorePlusQueries[PlusSandbox]Parser

[Christine Poerschke (JIRA)]

Christine Poerschke created LUCENE-6893:
---

 Summary: replace CorePlusExtensionsParser with 
CorePlusQueries[PlusSandbox]Parser
 Key: LUCENE-6893
 URL: https://issues.apache.org/jira/browse/LUCENE-6893
 Project: Lucene - Core
  Issue Type: Task
Reporter: Christine Poerschke
Assignee: Christine Poerschke
Priority: Minor


proposed change (patch against trunk to follow):
 * replace {{CorePlusExtensionsParser}} with {{CorePlusQueriesParser}} and 
{{CorePlusQueriesPlusSandboxParser}} (the latter extending the former).

motivation:
 * {{CorePlusExtensionsParser}} uses {{FuzzyLikeThisQueryBuilder}} which uses 
{{org.apache.lucene.sandbox.queries.(FuzzyLikeThisQuery|SlowFuzzyQuery)}}
 * we wish to use or inherit from {{CorePlusExtensionsParser}} but not pull in 
any sandbox code




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6892) various lucene.index initialCapacity tweaks

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000825#comment-15000825
 ] 

Christine Poerschke commented on LUCENE-6892:
-

note re: backporting to branch_5x: {{CodecReader.java}} has {{resources = new 
ArrayList<>(5);}} (5 instead of 6) since there is no "dimensional values" 
resource in 5x at present

> various lucene.index initialCapacity tweaks
> ---
>
> Key: LUCENE-6892
> URL: https://issues.apache.org/jira/browse/LUCENE-6892
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6892.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7569) Create an API to force a leader election between nodes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000726#comment-15000726
 ] 

ASF subversion and git services commented on SOLR-7569:
---

Commit 1713899 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1713899 ]

SOLR-7989, SOLR-7569: Ignore this test.

> Create an API to force a leader election between nodes
> --
>
> Key: SOLR-7569
> URL: https://issues.apache.org/jira/browse/SOLR-7569
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>  Labels: difficulty-medium, impact-high
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-7569-testfix.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569_lir_down_state_test.patch
>
>
> There are many reasons why Solr will not elect a leader for a shard e.g. all 
> replicas' last published state was recovery or due to bugs which cause a 
> leader to be marked as 'down'. While the best solution is that they never get 
> into this state, we need a manual way to fix this when it does get into this  
> state. Right now we can do a series of dance involving bouncing the node 
> (since recovery paths between bouncing and REQUESTRECOVERY are different), 
> but that is difficult when running a large cluster. Although it is possible 
> that such a manual API may lead to some data loss but in some cases, it is 
> the only possible option to restore availability.
> This issue proposes to build a new collection API which can be used to force 
> replicas into recovering a leader while avoiding data loss on a best effort 
> basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000725#comment-15000725
 ] 

ASF subversion and git services commented on SOLR-7989:
---

Commit 1713899 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1713899 ]

SOLR-7989, SOLR-7569: Ignore this test.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-7569) Create an API to force a leader election between nodes

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reopened SOLR-7569:
---

This feature and it's test counts on behavior that was added that was 
incorrect. See SOLR-7989. I've disabled the test for now.

> Create an API to force a leader election between nodes
> --
>
> Key: SOLR-7569
> URL: https://issues.apache.org/jira/browse/SOLR-7569
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>  Labels: difficulty-medium, impact-high
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-7569-testfix.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569_lir_down_state_test.patch
>
>
> There are many reasons why Solr will not elect a leader for a shard e.g. all 
> replicas' last published state was recovery or due to bugs which cause a 
> leader to be marked as 'down'. While the best solution is that they never get 
> into this state, we need a manual way to fix this when it does get into this  
> state. Right now we can do a series of dance involving bouncing the node 
> (since recovery paths between bouncing and REQUESTRECOVERY are different), 
> but that is difficult when running a large cluster. Although it is possible 
> that such a manual API may lead to some data loss but in some cases, it is 
> the only possible option to restore availability.
> This issue proposes to build a new collection API which can be used to force 
> replicas into recovering a leader while avoiding data loss on a best effort 
> basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7569) Create an API to force a leader election between nodes

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000753#comment-15000753
 ] 

Mark Miller commented on SOLR-7569:
---

A better approach is probably for this API to deal with a DOWN but valid leader 
itself. It should only ever happen due to manually screwing up LIR and if this 
API is messing with LIR, it should also fix the ramifications.

Perhaps the last thing the API should do is run through each shard and see if 
the registered leader is DOWN, and if it is make it ACTIVE (preferably by 
asking it to publish itself as ACTIVE - we don't want to publish for someone 
else). If the call waits around to make sure all the leaders come up, this 
should be simple.

> Create an API to force a leader election between nodes
> --
>
> Key: SOLR-7569
> URL: https://issues.apache.org/jira/browse/SOLR-7569
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>  Labels: difficulty-medium, impact-high
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-7569-testfix.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569_lir_down_state_test.patch
>
>
> There are many reasons why Solr will not elect a leader for a shard e.g. all 
> replicas' last published state was recovery or due to bugs which cause a 
> leader to be marked as 'down'. While the best solution is that they never get 
> into this state, we need a manual way to fix this when it does get into this  
> state. Right now we can do a series of dance involving bouncing the node 
> (since recovery paths between bouncing and REQUESTRECOVERY are different), 
> but that is difficult when running a large cluster. Although it is possible 
> that such a manual API may lead to some data loss but in some cases, it is 
> the only possible option to restore availability.
> This issue proposes to build a new collection API which can be used to force 
> replicas into recovering a leader while avoiding data loss on a best effort 
> basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8270) Make BM25SimFactory the implicit default when no sim is configured for luceneMatch > 6.0

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-8270.

Resolution: Fixed

> Make BM25SimFactory the implicit default when no sim is configured for 
> luceneMatch > 6.0
> 
>
> Key: SOLR-8270
> URL: https://issues.apache.org/jira/browse/SOLR-8270
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: Trunk
>
> Attachments: SOLR-8270.patch
>
>
> As discussed in parent issue, when the luceneMatchVersion is >= 6.0, 
> IndexSearcher should use BM25SimilarityFactory as the implicit default if no 
> explicit default is configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8057) Change default Sim to BM25 (w/backcompat config handling)

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man resolved SOLR-8057.

Resolution: Fixed

all subtasks completed

> Change default Sim to BM25 (w/backcompat config handling)
> -
>
> Key: SOLR-8057
> URL: https://issues.apache.org/jira/browse/SOLR-8057
> Project: Solr
>  Issue Type: Task
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Blocker
> Fix For: Trunk
>
> Attachments: SOLR-8057.patch, SOLR-8057.patch
>
>
> LUCENE-6789 changed the default similarity for IndexSearcher to BM25 and 
> renamed "DefaultSimilarity" to "ClassicSimilarity"
> Solr needs to be updated accordingly:
> * a "ClassicSimilarityFactory" should exist w/expected behavior/javadocs
> * default behavior (in 6.0) when no similarity is specified in configs should 
> (ultimately) use BM25 depending on luceneMatchVersion
> ** either by assuming BM25SimilarityFactory or by changing the internal 
> behavior of DefaultSimilarityFactory
> * comments in sample configs need updated to reflect new default behavior
> * ref guide needs updated anywhere it mentions/implies that a particular 
> similarity is used (or implies TF-IDF is used by default)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6893) replace CorePlusExtensionsParser with CorePlusQueries[PlusSandbox]Parser

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated LUCENE-6893:

Attachment: LUCENE-6893.patch

> replace CorePlusExtensionsParser with CorePlusQueries[PlusSandbox]Parser
> 
>
> Key: LUCENE-6893
> URL: https://issues.apache.org/jira/browse/LUCENE-6893
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6893.patch
>
>
> proposed change (patch against trunk to follow):
>  * replace {{CorePlusExtensionsParser}} with {{CorePlusQueriesParser}} and 
> {{CorePlusQueriesPlusSandboxParser}} (the latter extending the former).
> motivation:
>  * {{CorePlusExtensionsParser}} uses {{FuzzyLikeThisQueryBuilder}} which uses 
> {{org.apache.lucene.sandbox.queries.(FuzzyLikeThisQuery|SlowFuzzyQuery)}}
>  * we wish to use or inherit from {{CorePlusExtensionsParser}} but not pull 
> in any sandbox code



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 181 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/181/
Java: multiarch/jdk1.7.0 -d64 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.update.AutoCommitTest.testMaxDocs

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([49DF733F6981AC0D:F05EA5E0456BA887]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:766)
at 
org.apache.solr.update.AutoCommitTest.testMaxDocs(AutoCommitTest.java:194)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:qt=standard=0=20=2.2=id:14
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:759)
... 40 more




Build Log:
[...truncated 10346 lines...]
   [junit4] Suite: org.apache.solr.update.AutoCommitTest
   

[jira] [Commented] (SOLR-8225) Leader should send update requests to replicas in recovery asynchronously

[Ayon Sinha (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000681#comment-15000681
 ] 

Ayon Sinha commented on SOLR-8225:
--

[~markrmil...@gmail.com] "min replication factor " that sounds interesting. I 
didn't know SolrCloud could do that. I thought it was a CP i.e. consistency=all.
If it indeed is possible and we could make it dynamic such that min replication 
factor = num active nodes for that shard it would solve the problem.

> Leader should send update requests to replicas in recovery asynchronously
> -
>
> Key: SOLR-8225
> URL: https://issues.apache.org/jira/browse/SOLR-8225
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Reporter: Timothy Potter
>
> When a replica goes into recovery, the leader still sends docs to that 
> replica while it is recovering. What I'm seeing is that the recovering node 
> is still slow to respond to the leader (at least slower than the healthy 
> replicas). Thus it would be good if the leader could send the updates to the 
> recovering replica asynchronously, i.e. the leader will block as it does 
> today when forwarding updates to healthy / active replicas, but send updates 
> to recovering replicas async, thus preventing the whole update request from 
> being slowed down by a potentially degraded.
> FWIW - I've actually seen this occur in an environment that has more than 3 
> replicas per shard. One of the replicas went into recovery and then was much 
> slower to handle requests than the healthy replicas, but the leader had to 
> wait for the slowest replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-5819) Investigate & reduce size of ref-guide PDF

[Cassandra Targett (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett closed SOLR-5819.
---
Resolution: Fixed

This problem went away when Confluence was upgraded to v5.8.4.

> Investigate & reduce size of ref-guide PDF
> --
>
> Key: SOLR-5819
> URL: https://issues.apache.org/jira/browse/SOLR-5819
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
> Attachments: img-0007.png, img-0008.png, img-0009.png, img-0010.png, 
> img-0011.png, img-0012.png, img-0013.png, img-0014.png
>
>
> As noted on the solr-user mailing list in response to the ANNOUNCE about the 
> 4.7 ref guide, the size of the 4.4, 4.5 & 4.6 PDF files were all under 5MB, 
> but the 4.7 PDF was 30MB.
> We've determine that the root cause is a bug in confluence 5.0 (related to 
> duplicating images) that is fixed in 5.4.3 -- the next version Infra 
> currently plans to upgrade to.
> Until such time as the upgrade is finished, a work arround is to use a manual 
> pdf shrinking tool such as this one to eliminate the duplication...
> https://github.com/hossman/pdf-shrinker



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6892) various lucene.index initialCapacity tweaks

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000779#comment-15000779
 ] 

ASF subversion and git services commented on LUCENE-6892:
-

Commit 1713903 from [~cpoerschke] in branch 'dev/trunk'
[ https://svn.apache.org/r1713903 ]

LUCENE-6892: various lucene.index initialCapacity tweaks

> various lucene.index initialCapacity tweaks
> ---
>
> Key: LUCENE-6892
> URL: https://issues.apache.org/jira/browse/LUCENE-6892
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6892.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8225) Leader should send update requests to replicas in recovery asynchronously

[Ayon Sinha (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000674#comment-15000674
 ] 

Ayon Sinha commented on SOLR-8225:
--

Can a recovering replica after a hiccup behave like recovery after "add 
replica"? Or is it already the same mechanism? What are the downsides of leader 
sending updates to only active replicas and the recovering can catch-up as and 
when they can.

Usually replicas will go into recovery due to one of the following common 
reasons:
1. GC pause
2. brief Network outage 

> Leader should send update requests to replicas in recovery asynchronously
> -
>
> Key: SOLR-8225
> URL: https://issues.apache.org/jira/browse/SOLR-8225
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Reporter: Timothy Potter
>
> When a replica goes into recovery, the leader still sends docs to that 
> replica while it is recovering. What I'm seeing is that the recovering node 
> is still slow to respond to the leader (at least slower than the healthy 
> replicas). Thus it would be good if the leader could send the updates to the 
> recovering replica asynchronously, i.e. the leader will block as it does 
> today when forwarding updates to healthy / active replicas, but send updates 
> to recovering replicas async, thus preventing the whole update request from 
> being slowed down by a potentially degraded.
> FWIW - I've actually seen this occur in an environment that has more than 3 
> replicas per shard. One of the replicas went into recovery and then was much 
> slower to handle requests than the healthy replicas, but the leader had to 
> wait for the slowest replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7569) Create an API to force a leader election between nodes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000722#comment-15000722
 ] 

ASF subversion and git services commented on SOLR-7569:
---

Commit 1713898 from [~markrmil...@gmail.com] in branch 'dev/trunk'
[ https://svn.apache.org/r1713898 ]

SOLR-7989, SOLR-7569: Ignore this test.

> Create an API to force a leader election between nodes
> --
>
> Key: SOLR-7569
> URL: https://issues.apache.org/jira/browse/SOLR-7569
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>  Labels: difficulty-medium, impact-high
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-7569-testfix.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569_lir_down_state_test.patch
>
>
> There are many reasons why Solr will not elect a leader for a shard e.g. all 
> replicas' last published state was recovery or due to bugs which cause a 
> leader to be marked as 'down'. While the best solution is that they never get 
> into this state, we need a manual way to fix this when it does get into this  
> state. Right now we can do a series of dance involving bouncing the node 
> (since recovery paths between bouncing and REQUESTRECOVERY are different), 
> but that is difficult when running a large cluster. Although it is possible 
> that such a manual API may lead to some data loss but in some cases, it is 
> the only possible option to restore availability.
> This issue proposes to build a new collection API which can be used to force 
> replicas into recovering a leader while avoiding data loss on a best effort 
> basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000721#comment-15000721
 ] 

ASF subversion and git services commented on SOLR-7989:
---

Commit 1713898 from [~markrmil...@gmail.com] in branch 'dev/trunk'
[ https://svn.apache.org/r1713898 ]

SOLR-7989, SOLR-7569: Ignore this test.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-7989:
--
Attachment: SOLR-7989.patch

Something like this might work.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7569) Create an API to force a leader election between nodes

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000797#comment-15000797
 ] 

Mark Miller commented on SOLR-7569:
---

bq. It should only ever happen due to manually screwing up LIR and if this API 
is messing with LIR

Down the road though, we will want to solve this for SOLR-7034 and SOLR-7065.

I've taken a crack at making SOLR-7989 work.

> Create an API to force a leader election between nodes
> --
>
> Key: SOLR-7569
> URL: https://issues.apache.org/jira/browse/SOLR-7569
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>  Labels: difficulty-medium, impact-high
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-7569-testfix.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569.patch, SOLR-7569.patch, SOLR-7569.patch, 
> SOLR-7569_lir_down_state_test.patch
>
>
> There are many reasons why Solr will not elect a leader for a shard e.g. all 
> replicas' last published state was recovery or due to bugs which cause a 
> leader to be marked as 'down'. While the best solution is that they never get 
> into this state, we need a manual way to fix this when it does get into this  
> state. Right now we can do a series of dance involving bouncing the node 
> (since recovery paths between bouncing and REQUESTRECOVERY are different), 
> but that is difficult when running a large cluster. Although it is possible 
> that such a manual API may lead to some data loss but in some cases, it is 
> the only possible option to restore availability.
> This issue proposes to build a new collection API which can be used to force 
> replicas into recovering a leader while avoiding data loss on a best effort 
> basis.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8274) Add per-request MDC logging based on user-provided value.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000820#comment-15000820
 ] 

Mark Miller commented on SOLR-8274:
---

bq. especially when we need to jump thread/process boundaries

interesting Apahce project along those lines: http://htrace.org/

> Add per-request MDC logging based on user-provided value.
> -
>
> Key: SOLR-8274
> URL: https://issues.apache.org/jira/browse/SOLR-8274
> Project: Solr
>  Issue Type: Improvement
>Reporter: Jason Gerlowski
>Priority: Minor
>
> *Problem 1* Currently, there's no way (AFAIK) to find all log messages 
> associated with a particular request.
> *Problem 2* There's also no easy way for multi-tenant Solr setups to find all 
> log messages associated with a particular customer/tenant.
> Both of these problems would be more manageable if Solr could be configured 
> to record an MDC tag based on a header, or some other user provided value.
> This would allow admins to group together logs about a single request.  If 
> the same header value is repeated multiple times this functionality could 
> also be used to group together arbitrary requests, such as those that come 
> from a particular user, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-8075:
--
Attachment: SOLR-8075.patch

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8270) Make BM25SimFactory the implicit default when no sim is configured for luceneMatch > 6.0

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000773#comment-15000773
 ] 

ASF subversion and git services commented on SOLR-8270:
---

Commit 1713902 from hoss...@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1713902 ]

SOLR-8270: Change implicit default Similarity to use BM25 when 
luceneMatchVersion >= 6

> Make BM25SimFactory the implicit default when no sim is configured for 
> luceneMatch > 6.0
> 
>
> Key: SOLR-8270
> URL: https://issues.apache.org/jira/browse/SOLR-8270
> Project: Solr
>  Issue Type: Sub-task
>Reporter: Hoss Man
>Assignee: Hoss Man
> Fix For: Trunk
>
> Attachments: SOLR-8270.patch
>
>
> As discussed in parent issue, when the luceneMatchVersion is >= 6.0, 
> IndexSearcher should use BM25SimilarityFactory as the implicit default if no 
> explicit default is configured.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7951) LBHttpSolrClient wraps ALL exceptions in "No live SolrServers available to handle this request" exception, even usage errors

[Elaine Cario (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000782#comment-15000782
 ] 

Elaine Cario commented on SOLR-7951:


NP - we applied patch to 4.10 in our prod environment some weeks ago, so far 
have seen no issues (and it fixed the issue we were having).  But it would be 
good to get it in 5.x so I don't need to worry about carrying the patch forward 
when we upgrade (no date for that, but it's in the planning stages).

Thanks for the follow-up!

> LBHttpSolrClient wraps ALL exceptions in "No live SolrServers available to 
> handle this request" exception, even usage errors
> 
>
> Key: SOLR-7951
> URL: https://issues.apache.org/jira/browse/SOLR-7951
> Project: Solr
>  Issue Type: Bug
>  Components: SolrJ
>Affects Versions: 5.2.1
>Reporter: Elaine Cario
>Assignee: Mark Miller
>Priority: Minor
> Attachments: SOLR-7951-4.x.patch, SOLR-7951-4.x.patch, SOLR-7951.patch
>
>
> We were experiencing many "No live SolrServers available to handle this 
> request" exception, even though we saw no outages with any of our servers.  
> It turned out the actual exceptions were related to the use of wildcards in 
> span queries (and in some cases other invalid queries or usage-type issues). 
> Traced it back to LBHttpSolrClient which was wrapping all exceptions, even 
> plain SolrExceptions, in that outer exception.  
> Instead, wrapping in the out exception should be reserved for true 
> communication issues in SolrCloud, and usage exceptions should be thrown as 
> is.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Windows (64bit/jdk1.8.0_66) - Build # 5392 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Windows/5392/
Java: 64bit/jdk1.8.0_66 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
at 
__randomizedtesting.SeedInfo.seed([8AF5417FD17E023:EE3860D7C4951942]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:170)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
at 
org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
  

[JENKINS-EA] Lucene-Solr-trunk-Linux (32bit/jdk1.9.0-ea-b90) - Build # 14871 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/14871/
Java: 32bit/jdk1.9.0-ea-b90 -client -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader

Error Message:
There are still nodes recoverying - waited for 330 seconds

Stack Trace:
java.lang.AssertionError: There are still nodes recoverying - waited for 330 
seconds
at 
__randomizedtesting.SeedInfo.seed([5A84B20C2E2D0B8:E33F7FE0FB6029D9]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:170)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:131)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:126)
at 
org.apache.solr.cloud.ForceLeaderTest.testReplicasInLIRNoLeader(ForceLeaderTest.java:115)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:520)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 

[jira] [Commented] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000465#comment-15000465
 ] 

Mark Miller commented on SOLR-8075:
---

Hmm...this still needs a little tweaking I think.

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reassigned SOLR-7989:
-

Assignee: Mark Miller  (was: Noble Paul)

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000474#comment-15000474
 ] 

Mark Miller commented on SOLR-7989:
---

I actually think this is wrong on other levels.

All publishing of state should go through ZkController - we do things like 
track the last published state. I'm not sure this is the right place to do this 
either. I'm reviewing this in another issue.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Solaris (64bit/jdk1.8.0) - Build # 181 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Solaris/181/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.update.HardAutoCommitTest.testCommitWithin

Error Message:
Exception during query

Stack Trace:
java.lang.RuntimeException: Exception during query
at 
__randomizedtesting.SeedInfo.seed([AAB597A0372DADDD:1067F8D8B40343C8]:0)
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:766)
at 
org.apache.solr.update.HardAutoCommitTest.testCommitWithin(HardAutoCommitTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:54)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:55)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: REQUEST FAILED: 
xpath=//result[@numFound=1]
xml response was: 

00


request was:q=id:529=standard=0=20=2.2
at org.apache.solr.SolrTestCaseJ4.assertQ(SolrTestCaseJ4.java:759)
... 40 more




Build Log:
[...truncated 10101 lines...]
   [junit4] Suite: 

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000483#comment-15000483
 ] 

Mark Miller commented on SOLR-7989:
---

Also, what about basic stuff like:

Core starts up, starts election, it's not active. Becomes leader, it's not 
active so it's made active with this change.

Now register is called. Register has the core do tlog replay and then is ready 
to go active. So it does. But we are already active too early?

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000571#comment-15000571
 ] 

ASF subversion and git services commented on SOLR-7989:
---

Commit 1713882 from [~markrmil...@gmail.com] in branch 'dev/trunk'
[ https://svn.apache.org/r1713882 ]

SOLR-7989: Revert changes.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Reopened] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller reopened SOLR-7989:
---

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000554#comment-15000554
 ] 

Mark Miller commented on SOLR-7989:
---

Hmm...doesn't this test create the problem itself?

It puts all bunch of replicas in LIR manually, causing the DOWN state. Then it 
manually clears LIR and tries to see who will become leader.

This is not a normal case?

In any case, I will revert the current fix. Now I think the test may be 
creating the problem itself, but regardless, this change really breaks things.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-8075:
--
Attachment: SOLR-8075.patch

New patch.

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-6892) various lucene.index initialCapacity tweaks

[Simon Willnauer (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000341#comment-15000341
 ] 

Simon Willnauer commented on LUCENE-6892:
-

LGTM

> various lucene.index initialCapacity tweaks
> ---
>
> Key: LUCENE-6892
> URL: https://issues.apache.org/jira/browse/LUCENE-6892
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: LUCENE-6892.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-trunk-Linux (64bit/jdk1.8.0_66) - Build # 14869 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/14869/
Java: 64bit/jdk1.8.0_66 -XX:-UseCompressedOops -XX:+UseParallelGC

3 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CollectionsAPIDistributedZkTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 1) Thread[id=933, 
name=zkCallback-36-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)2) Thread[id=258, 
name=zkCallback-36-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)3) Thread[id=256, 
name=TEST-CollectionsAPIDistributedZkTest.test-seed#[5C9BBE9138FC6616]-SendThread(127.0.0.1:48848),
 state=TIMED_WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at 
java.lang.Thread.sleep(Native Method) at 
org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:994)4) 
Thread[id=257, 
name=TEST-CollectionsAPIDistributedZkTest.test-seed#[5C9BBE9138FC6616]-EventThread,
 state=WAITING, group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
 at 
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) 
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:494)
5) Thread[id=932, name=zkCallback-36-thread-2, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest] at 
sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) 
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
 at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
 at 
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941) 
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.CollectionsAPIDistributedZkTest: 
   1) Thread[id=933, name=zkCallback-36-thread-3, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at 
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
at 
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1066)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
   2) Thread[id=258, name=zkCallback-36-thread-1, state=TIMED_WAITING, 
group=TGRP-CollectionsAPIDistributedZkTest]
at 

Re: Inconsistent error returned by Solr when indexing bad documents

[Mark Miller]

Yeah, the error handling is the same regardless. The error handling code is
tricky to follow because of batch and streaming - but they dont have
special handling vs single update.

- Mark

On Wed, Nov 11, 2015 at 1:00 AM Shai Erera  wrote:

> OK just wanted to clarify that this happens even when indexing a single
> document, using curl or SolrJ:
>
> curl -i -X POST http://localhost:7574/solr/gettingstarted/update/json/docs
> -d '{"id" : "1", "unknown" : "foo"}'
>
> So it's not only with bulk updates.
>
> I will nevertheless reproduce this in a proper unit test.
>
> Shai
>
> On Tue, Nov 10, 2015 at 6:31 PM, Mark Miller 
> wrote:
>
>> It's kind of tricky. And it doesn't help that we don't have real per
>> update errors when doing bulk or streaming.
>>
>> But you can start looking around DistributedUpdateProcessor,
>> SolrCmdDistributor, StreamingSolrClients for update error propagation
>> 'stuff'.
>>
>> - Mark
>>
>> On Tue, Nov 10, 2015 at 10:39 AM Shai Erera  wrote:
>>
>>> Thanks Mark, I wrote a test for it, I can port it to Solr's
>>> test-framework. Can you also give me a hint in what area of the code I
>>> should look to fix it?
>>>
>>> Shai
>>>
>>> On Tue, Nov 10, 2015 at 4:59 PM, Mark Miller 
>>> wrote:
>>>
 It's not properly propagating the root error it loos. We probably need
 a test for that.

 - Mark

 On Tue, Nov 10, 2015 at 7:54 AM Shai Erera  wrote:

> Hi,
>
> I wanted to test the error message that Solr returns when indexing a
> document with an unknown field. Surprisingly, I get different errors,
> depending if the request hits the shard's leader or not.
>
> To reproduce (5.3.1):
>
> bin/solr -e cloud
>   ports: 8983, 7574
>   config: basic_configs
>   shards: 1
>   replicas: 2
>
> Wait for the nodes to come up and issue a CLUSTERSTATUS call to check
> which replica is the leader. In my case, 7574 was the leader. Now
> index a document with an unknown field:
>
> curl -i -X POST http://localhost:8983/solr/gettingstarted/update/json
> -d '[{"id" : "1", "unknown" : "foo"}]'
>
> And you get back
>
> {"responseHeader":{"status":400,"QTime":6},"error":{"msg":"Bad
> Request\n\n\n\nrequest:
> http://169.254.21.228:7574/solr/gettingstarted_shard1_replica1/update?update.distrib=TOLEADER=http%3A%2F%2F169.254.21.228%3A8983%2Fsolr%2Fgettingstarted_shard1_replica2%2F=javabin=2
> ","code":400}}
>
> But if you execute:
>
> curl -i -X POST http://localhost:7574/solr/gettingstarted/update/json
> -d '[{"id" : "1", "unknown" : "foo"}]'
>
> Then you get back
>
> {"responseHeader":{"status":400,"QTime":1},"error":{"msg":"ERROR:
> [doc=1] unknown field 'unknown'","code":400}}
>
> In both cases you get back 400, but if the request hits the leader you
> get a more expressive error message. Is there any reason for that 
> behavior?
> Can't the replica just pass along the error that it got from the leader?
>
>
> Shai
>
 --
 - Mark
 about.me/markrmiller

>>>
>>> --
>> - Mark
>> about.me/markrmiller
>>
>
> --
- Mark
about.me/markrmiller

[jira] [Commented] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000452#comment-15000452
 ] 

Mark Miller commented on SOLR-8075:
---

Patch with fix.

bq. How can I investigate / reproduce this?

Don't know. Saw it not working on a real cluster.

I also saw the shard get stuck with no proper leader still due to the proper 
leader being in LIR. It seems LIR is still a little buggy. Fixing this should 
be one step towards alleviating that.

I'm going to work on a test that indexes while restarting the whole cluster and 
see if I can reproduce some of those bugs.

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000486#comment-15000486
 ] 

Mark Miller commented on SOLR-7989:
---

There is def an issue to be fixed here, but I think it may be more complicated 
than this.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000574#comment-15000574
 ] 

ASF subversion and git services commented on SOLR-7989:
---

Commit 1713883 from [~markrmil...@gmail.com] in branch 'dev/branches/branch_5x'
[ https://svn.apache.org/r1713883 ]

SOLR-7989: Revert changes.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000576#comment-15000576
 ] 

Mark Miller commented on SOLR-7989:
---

[~ichattopadhyaya], I don't think this is a valid test at all.

Why does it clear LIR while Solr is running? That is not a valid real world 
scenario.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-8075:
--
Attachment: SOLR-8075.patch

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch, SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-8069) Ensure that only the valid ZooKeeper registered leader can put a replica into Leader Initiated Recovery.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller resolved SOLR-8069.
---
Resolution: Fixed

> Ensure that only the valid ZooKeeper registered leader can put a replica into 
> Leader Initiated Recovery.
> 
>
> Key: SOLR-8069
> URL: https://issues.apache.org/jira/browse/SOLR-8069
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Critical
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8069.patch, SOLR-8069.patch
>
>
> I've seen this twice now. Need to work on a test.
> When some issues hit all the replicas at once, you can end up in a situation 
> where the rightful leader was put or put itself into LIR. Even on restart, 
> this rightful leader won't take leadership and you have to manually clear the 
> LIR nodes.
> It seems that if all the replicas participate in election on startup, LIR 
> should just be cleared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8069) Ensure that only the valid ZooKeeper registered leader can put a replica into Leader Initiated Recovery.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000453#comment-15000453
 ] 

Mark Miller commented on SOLR-8069:
---

The problem is unrelated to this issue. That assert is correct and it's 
catching a bug with SOLR-8075 or something. It's passing null when it should 
pass the core descriptor.

> Ensure that only the valid ZooKeeper registered leader can put a replica into 
> Leader Initiated Recovery.
> 
>
> Key: SOLR-8069
> URL: https://issues.apache.org/jira/browse/SOLR-8069
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Critical
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8069.patch, SOLR-8069.patch
>
>
> I've seen this twice now. Need to work on a test.
> When some issues hit all the replicas at once, you can end up in a situation 
> where the rightful leader was put or put itself into LIR. Even on restart, 
> this rightful leader won't take leadership and you have to manually clear the 
> LIR nodes.
> It seems that if all the replicas participate in election on startup, LIR 
> should just be cleared.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000578#comment-15000578
 ] 

Mark Miller commented on SOLR-7989:
---

bq. Why does it clear LIR while Solr is running? 

I guess this tests our emergency force a leader API? That's part of why that 
API is so janky. It's not really valid to simply clear out LIR nodes on a 
running system. And it's a really hard problem to make it doable - this 
approach certainly won't work.

What we really need to do is fix things so it's not needed.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  \{!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  \{!djoin\}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler 

[jira] [Updated] (SOLR-8234) Federated Search (new) - DJoin

[Tom Winch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tom Winch updated SOLR-8234:

Description: 
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents, with an indicator field in the parent - see example 
output, below).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older issues SOLR-3799 through SOLR-3805.

--

Example request handler configuration:

{code:xml}
  
djoin
  
  
  

  


  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  {!djoin}


  filter

   
{code}

Example output:

{code:xml}


  
0
33

  *:*
  http://shard1/solr/core,http://shard2/solr/core,http://shard3/solr/core
  true
  xml
  {!djoin}
  *,[shard]

  
  

  true
  
200
1973
http://shard2/solr/core
1515645309629235200
  
  
200
2015
http://shard1/solr/core
1515645309682712576
  


  true
  
100
873
http://shard1/solr/core
1515645309629254124
  
  
100
2001
http://shard3/solr/core
1515645309682792852
  


  true
  
300
1492
http://shard2/solr/core
1515645309629251252
  

  

{code}

  was:
This issue describes a MergeStrategy implementation (DJoin) to facilitate 
federated search - that is, distributed search over documents stored in 
separated instances of SOLR (for example, one server per continent), where a 
single document (identified by an agreed, common unique id) may be stored in 
more than one server instance, with (possibly) differing fields and data.

When the MergeStrategy is used in a request handler (via the included QParser) 
in combination with distributed search (shards=), documents having an id that 
has already been seen are not discarded (as per the default behaviour) but, 
instead, are collected and returned as a group of documents all with the same 
id taking a single position in the result set (this is implemented using 
parent/child documents).

Documents are sorted in the result set based on the highest ranking document 
with the same id. It is possible for a document ranking high in one shard to 
rank very low on another shard. As a consequence of this, all shards must be 
asked to return the fields for every document id in the result set (not just of 
those documents they returned), so that all the component parts of each 
document in the search result set are returned.

As usual, search parameters are passed on to each shard. So that the shards do 
not need any additional configurations in their definition of the /select 
request handler, we use the FilterQParserSearchComponent which is configured to 
filter out the \{!djoin\} search parameter - otherwise, the target request 
handler complains about the missing query parser definition. See the example 
config, below.

This issue combines with others to provide full federated search support. See 
also SOLR-8235 and SOLR-8236.

Note that this is part of a new implementation of federated search as opposed 
to the older 

[jira] [Created] (LUCENE-6891) Lucene30DimensionalFormat should use block prefix coding when writing values

[Michael McCandless (JIRA)]

Michael McCandless created LUCENE-6891:
--

 Summary: Lucene30DimensionalFormat should use block prefix coding 
when writing values
 Key: LUCENE-6891
 URL: https://issues.apache.org/jira/browse/LUCENE-6891
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Michael McCandless
Assignee: Michael McCandless
 Fix For: Trunk


Today we write the whole value for every doc in one leaf block in the BKD tree, 
but that's crazy because the whole point of that leaf block is all the docs 
inside it have values that are very close together.

So I changed this to write the common prefix for the whole block up front in 
each block.  This requires more index-time and search-time work, but gives nice 
index size reductions:

On the 2D (London, UK) lat/lon benchmark:
  * Indexing time was a wee bit slower (743 -> 747 seconds)
  * Index size was ~11% smaller (704 MB -> 630 MB)
  * Query time was ~7% slower (2.84 sec -> 3.05 sec)
  * Heap usage is the same

On the 1D (just "lat" from the above test) benchmark:
  * Indexing time was a wee bit slower (363 -> 364 sec)
  * Index size was ~23% smaller (472 MB -> 363 MB)
  * Query time was a wee bit slower (5.39 -> 5.41 sec)
  * Heap usage is the same

Index time can be a bit slower since there are two passes now per leaf block 
(first to find the common prefix per dimension, and second pass must then strip 
those prefixes).

Query time is slower because there's more work per hit that needs value 
filtering, i.e. collating the suffixes onto the prefixes, per dimension.  This 
affects 2D much more than 1D because 1D has fewer leaf blocks that need 
filtering (typically 0, 1 or 2, unless there are many duplicate values in the 
index).

I suspect the index size savings is use-case dependent, e.g. if you index a 
bunch of ipv4 addresses along with a few ipv6 addresses, you'd probably see 
sizable savings.

I also suspect the more docs you index, the greater the savings, because the 
cells will generally be smaller.

Net/net I think the opto is worth it, even if it slows query time a bit.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Windows (64bit/jdk1.7.0_80) - Build # 5261 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Windows/5261/
Java: 64bit/jdk1.7.0_80 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.cloud.CustomCollectionTest.test

Error Message:
Error from server at http://127.0.0.1:53526/_vl: collection already exists: 
routeFieldColl

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:53526/_vl: collection already exists: 
routeFieldColl
at 
__randomizedtesting.SeedInfo.seed([F322520CEF79F7EF:7B766DD641859A17]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:575)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:241)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:230)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:372)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:325)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1099)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:870)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1573)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.createCollection(AbstractFullDistribZkTestBase.java:1527)
at 
org.apache.solr.cloud.CustomCollectionTest.testRouteFieldForHashRouter(CustomCollectionTest.java:333)
at 
org.apache.solr.cloud.CustomCollectionTest.test(CustomCollectionTest.java:96)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 

[jira] [Created] (SOLR-8277) (Search|Top)GroupsFieldCommand tweaks

[Christine Poerschke (JIRA)]

Christine Poerschke created SOLR-8277:
-

 Summary: (Search|Top)GroupsFieldCommand tweaks
 Key: SOLR-8277
 URL: https://issues.apache.org/jira/browse/SOLR-8277
 Project: Solr
  Issue Type: Task
Reporter: Christine Poerschke
Assignee: Christine Poerschke
Priority: Minor


patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8277) (Search|Top)GroupsFieldCommand tweaks

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-8277:
--
Attachment: SOLR-8277.patch

proposed patch against trunk

> (Search|Top)GroupsFieldCommand tweaks
> -
>
> Key: SOLR-8277
> URL: https://issues.apache.org/jira/browse/SOLR-8277
> Project: Solr
>  Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-8277.patch
>
>
> patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-MacOSX (64bit/jdk1.7.0) - Build # 2808 - Failure!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-MacOSX/2808/
Java: 64bit/jdk1.7.0 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.CollectionsAPIDistributedZkTest.test

Error Message:
Error from server at http://127.0.0.1:64005/so_/wu/awholynewcollection_0: 
Expected mime type application/octet-stream but got text/html.   
 
Error 500HTTP ERROR: 500 Problem 
accessing /so_/wu/awholynewcollection_0/select. Reason: {msg=Error 
trying to proxy request for url: 
http://127.0.0.1:64024/so_/wu/awholynewcollection_0/select,trace=org.apache.solr.common.SolrException:
 Error trying to proxy request for url: 
http://127.0.0.1:64024/so_/wu/awholynewcollection_0/select  at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:596)  at 
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at 
org.eclipse.jetty.servlets.UserAgentFilter.doFilter(UserAgentFilter.java:83)  
at org.eclipse.jetty.servlets.GzipFilter.doFilter(GzipFilter.java:364)  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585) 
 at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)  
at org.eclipse.jetty.server.Server.handle(Server.java:499)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)  at 
org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555) 
 at java.lang.Thread.run(Thread.java:745) Caused by: 
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for 
connection from pool  at 
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection(PoolingClientConnectionManager.java:226)
  at 
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection(PoolingClientConnectionManager.java:195)
  at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
  at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
  at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
  at org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:563)  
... 24 more ,code=500} Powered by 
Jetty://   

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:64005/so_/wu/awholynewcollection_0: Expected 
mime type application/octet-stream but got text/html. 


Error 500 


HTTP ERROR: 500
Problem accessing /so_/wu/awholynewcollection_0/select. Reason:
{msg=Error trying to proxy request for url: 
http://127.0.0.1:64024/so_/wu/awholynewcollection_0/select,trace=org.apache.solr.common.SolrException:
 Error trying to proxy request for url: 
http://127.0.0.1:64024/so_/wu/awholynewcollection_0/select
at 
org.apache.solr.servlet.HttpSolrCall.remoteQuery(HttpSolrCall.java:596)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:444)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:220)
at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:179)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:109)
at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
at 

[jira] [Commented] (SOLR-8225) Leader should send update requests to replicas in recovery asynchronously

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000442#comment-15000442
 ] 

Yonik Seeley commented on SOLR-8225:


bq. What I'm seeing is that the recovering node is still slow to respond to the 
leader (at least slower than the healthy replicas).

Hmmm, that's interesting.  Any pointers as to why?  There's actually a lot less 
work (we just buffer in the tlog).
Perhaps it's the IO bandwidth being taken up by the index replication?

Sending async will introduce some complexities around the replica becoming 
active. Right now, the replica itself knows when it can become active... after 
it's finished replicating the index + replaying all buffered updates.  With an 
async-send, that would no longer be the case.

> Leader should send update requests to replicas in recovery asynchronously
> -
>
> Key: SOLR-8225
> URL: https://issues.apache.org/jira/browse/SOLR-8225
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Reporter: Timothy Potter
>
> When a replica goes into recovery, the leader still sends docs to that 
> replica while it is recovering. What I'm seeing is that the recovering node 
> is still slow to respond to the leader (at least slower than the healthy 
> replicas). Thus it would be good if the leader could send the updates to the 
> recovering replica asynchronously, i.e. the leader will block as it does 
> today when forwarding updates to healthy / active replicas, but send updates 
> to recovering replicas async, thus preventing the whole update request from 
> being slowed down by a potentially degraded.
> FWIW - I've actually seen this occur in an environment that has more than 3 
> replicas per shard. One of the replicas went into recovery and then was much 
> slower to handle requests than the healthy replicas, but the leader had to 
> wait for the slowest replica.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8278) ConfigSetService should use NIO2

[Alan Woodward (JIRA)]

Alan Woodward created SOLR-8278:
---

 Summary: ConfigSetService should use NIO2 
 Key: SOLR-8278
 URL: https://issues.apache.org/jira/browse/SOLR-8278
 Project: Solr
  Issue Type: Improvement
Reporter: Alan Woodward
Assignee: Alan Woodward


Similar to SOLR-8260, using java.nio.file.Path instead of String and File makes 
error handling and reporting a lot easier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8075) Leader Initiated Recovery should not stop a leader that participated in an election with all of it's replicas from becoming a valid leader.

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000428#comment-15000428
 ] 

Ishan Chattopadhyaya commented on SOLR-8075:


How can I investigate / reproduce this? Is the 
LeaderInitiatedRecoveryOnShardRestartTest failing under certain circumstances?

> Leader Initiated Recovery should not stop a leader that participated in an 
> election with all of it's replicas from becoming a valid leader.
> ---
>
> Key: SOLR-8075
> URL: https://issues.apache.org/jira/browse/SOLR-8075
> Project: Solr
>  Issue Type: Bug
>Reporter: Mark Miller
>Assignee: Mark Miller
> Fix For: 5.4, Trunk
>
> Attachments: SOLR-8075.patch, SOLR-8075.patch, SOLR-8075.patch, 
> SOLR-8075.patch
>
>
> Currently, because of SOLR-8069, all the replicas in a shard can be put into 
> LIR.
> If you restart such a shard, the valid leader will will win the election and 
> sync with the shard and then be blocked from registering as ACTIVE because it 
> is in LIR.
> I think that is a little wonky because I don't think it even tries another 
> candidate because the leader that cannot publish ACTIVE does not have it's 
> election canceled.
> While SOLR-8069 should prevent this situation, we should add logic to allow a 
> leader that can sync with it's full shard to become leader and publish ACTIVE 
> regardless of LIR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7989) Down replica elected leader, stays down after successful election

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15000414#comment-15000414
 ] 

Ishan Chattopadhyaya commented on SOLR-7989:


bq. you can do a force read before checking the state .
That's what I think I've done in the last patch. Can you/Mark please review?
Sorry for missing this out; I did this update cluster state in a patch before 
the original commit itself; but the LeaderElectionTest failure threw me off and 
I somehow missed the update cluster state in the final patch for the commit.

> Down replica elected leader, stays down after successful election
> -
>
> Key: SOLR-7989
> URL: https://issues.apache.org/jira/browse/SOLR-7989
> Project: Solr
>  Issue Type: Bug
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
> Fix For: 5.4, Trunk
>
> Attachments: DownLeaderTest.java, DownLeaderTest.java, 
> SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, SOLR-7989.patch, 
> SOLR-7989.patch, SOLR-8233.patch
>
>
> It is possible that a down replica gets elected as a leader, and that it 
> stays down after the election.
> Here's how I hit upon this:
> * There are 3 replicas: leader, notleader0, notleader1
> * Introduced network partition to isolate notleader0, notleader1 from leader 
> (leader puts these two in LIR via zk).
> * Kill leader, remove partition. Now leader is dead, and both of notleader0 
> and notleader1 are down. There is no leader.
> * Remove LIR znodes in zk.
> * Wait a while, and there happens a (flawed?) leader election.
> * Finally, the state is such that one of notleader0 or notleader1 (which were 
> down before) become leader, but stays down.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-6892) various lucene.index initialCapacity tweaks

[Christine Poerschke (JIRA)]

Christine Poerschke created LUCENE-6892:
---

 Summary: various lucene.index initialCapacity tweaks
 Key: LUCENE-6892
 URL: https://issues.apache.org/jira/browse/LUCENE-6892
 Project: Lucene - Core
  Issue Type: Task
Reporter: Christine Poerschke
Assignee: Christine Poerschke
Priority: Minor


patch to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8278) ConfigSetService should use NIO2

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated SOLR-8278:

Attachment: SOLR-8278.patch

Patch.  As well as changing ConfigSetService, this also alters NodeConfig to 
return Path objects from getCoreRootDirectory() and getConfigSetBaseDirectory()

> ConfigSetService should use NIO2 
> -
>
> Key: SOLR-8278
> URL: https://issues.apache.org/jira/browse/SOLR-8278
> Project: Solr
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
> Attachments: SOLR-8278.patch
>
>
> Similar to SOLR-8260, using java.nio.file.Path instead of String and File 
> makes error handling and reporting a lot easier.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Sharing a class across cores

[Gus Heck]

Thanks for the links and perspective Hoss 3443 is exactly the same type of
problem only for spellchecking. My initial thought was also a map, perhaps
I'll adopt that ticket. The basic idea would be a Map on
CoreContainer, accessors, and some very very strongly worded javadoc about
keeping things simple to avoid classloader memory leaks.

A slightly safer approach that would be much harder to implement would
involve counting references to the shared stuff so that if all cores using
it were removed, it could be cleaned up... Not sure if that's worth it.

-Gus

On Wed, Nov 11, 2015 at 7:17 PM, Chris Hostetter 
wrote:

>
> : However, my take on it is that this seems like a pretty broad brush to
> : paint with to move *all* our classes up and out of the normal core
> loading
> : process. I assume there are good reasons for segregating this stuff into
> : separate class loaders to begin with. It would also be fairly burdensom
> to
>
> There are, but those reasons don't really apply if the whole point is you
> want to share resources between cores.
>
> : I really just want a way to stash the map in a place where other cores
> can
> : see it (and thus I can appropriately synchronize things so that the
> loading
> : only happens once). I'm asking because it seems like surely this must be
> a
> : solved problem... if not, it might be easiest to just solve it by adding
> : some sort of shared resources facility to CoreContainer?
>
> There has been some discussion about it in the past (ie: multiple
> instances of StopwordFilterFactory configured to point at the same
> stopwords.txt file on disk can/should share the same Map in RAM (in most
> cases) even if those instances exist in completely diff cores)
>
> There's been a few jiras where this concept of "sharing" heavy objects
> have come up, but i don't think anyone has made any attempts at a general
> solution...
>
>
> https://issues.apache.org/jira/browse/SOLR-7282
>
> https://issues.apache.org/jira/browse/SOLR-4872?focusedCommentId=13682471=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13682471
> https://issues.apache.org/jira/browse/SOLR-3443
>
>
> -Hoss
> http://www.lucidworks.com/
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>


-- 
http://www.the111shift.com

[jira] [Updated] (SOLR-8280) TestCloudSchemaless fails weirdly if you try to use SolrCoreAware sim factory: SchemaSimilarityFactory

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-8280:
---
Attachment: SOLR-8280.patch


With the attached patch, it's trivial to trigger a suite failure in 
TestCloudSchemaless, although lots of other cloud related tests (using 
schema.xml via AbstractFullDistribZkTestBase) function just fine.

(NOTE: there are in fact a handful of other test failures with this patch, but 
they exist due to specific Sim related assumptions in those tests vs the 
general change of schema.xml to using SchemaSimilarityFactory ... i included 
the change to schema.xml in this patch purely as an easy way to demonstrate 
that most cloud related tests -- even ones using schemaless and/or involving 
core reloads -- don't ffreak out the same way as TestCloudSchemaless)



Quick walk through of the TestCloudSchemaless failure...

{noformat}
ant test  -Dtestcase=TestCloudSchemaless -Dtests.seed=8F1B744FBE000231 
-Dtests.slow=true
{noformat}

The test itself runs fine, and reaches the {{SolrTestCaseJ4 ###Ending test" 
stage}} at which point shutdown begins -- and eventually stalls in the dir 
factory waiting for directories to close...

{noformat}
   [junit4]   2> 52621 INFO  
(TEST-TestCloudSchemaless.test-seed#[E3CE46BC04E096EF]) [n:127.0.0.1:35554_ 
c:collection1 s:shard2 r:core_node1 x:collection1] o.a.s.c.SolrCore 
[collection1] Closing main searcher on request.
   [junit4]   2> 52621 INFO  
(TEST-TestCloudSchemaless.test-seed#[E3CE46BC04E096EF]) [n:127.0.0.1:35554_ 
c:collection1 s:shard2 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory Closing NRTCachingDirectoryFactory - 2 
directories currently being tracked
   [junit4]   2> 64733 ERROR 
(TEST-TestCloudSchemaless.test-seed#[E3CE46BC04E096EF]) [n:127.0.0.1:35554_ 
c:collection1 s:shard2 r:core_node1 x:collection1] 
o.a.s.c.CachingDirectoryFactory Timeout waiting for all directory ref counts to 
be released - gave up waiting on 

[jira] [Updated] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-6874:
--
Attachment: (was: unicode-ws-tokenizer.patch)

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-6874:
--
Attachment: unicode-ws-tokenizer.patch

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch, 
> unicode-ws-tokenizer.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-6874:
--
Attachment: (was: unicode-ws-tokenizer.patch)

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8280) TestCloudSchemaless fails weirdly if you try to use SolrCoreAware sim factory: SchemaSimilarityFactory

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-8280:
---
Attachment: SOLR-8280.patch

After filing this, it occurred to me the original issue that lead me down this 
investigation (SOLR-8271) I was also seeing failures in 
ChangedSchemaMergeTest.testOptimizeDiffSchemas that i wasn't seeing with this 
patch.  Looking into why I realized it's because that test uses it's own 
hardcoded schema(s) that are build on the fly from java code.

In this new updated patch, i tweaked them to directly refer to 
SchemaSimilarityFactory, and now you can see the {{AssertionError: inform must 
be called first}} getting logged, and the subsequent cc.shutdown() giving up 
after a ref count mismatch on the directory factory.

This is probably a simpler test to trouble shoot why the SolrCoreAware.inform 
isn't getting called properly then the cloud test, and hopefully can lead to a 
fix that helps with both tests.

> TestCloudSchemaless fails weirdly if you try to use SolrCoreAware sim 
> factory: SchemaSimilarityFactory 
> ---
>
> Key: SOLR-8280
> URL: https://issues.apache.org/jira/browse/SOLR-8280
> Project: Solr
>  Issue Type: Bug
>Affects Versions: Trunk
>Reporter: Hoss Man
> Attachments: SOLR-8280.patch, SOLR-8280.patch
>
>
> Something about the code path(s) involved in TestCloudSchemaless doesn't play 
> nicely with a SimilarityFactory that is SolrCoreAware -- notably: 
> SchemaSimilarityFactory.
> I discovered this while trying to implement SOLR-8271, but it can be 
> reproduced trivially by modifying the 
> schema-add-schema-fields-update-processor.xml file used by 
> TestCloudSchemaless to refer to SchemaSimilarityFactory explicitly.  Other 
> cloud tests (such as CollectionReloadTest) or cloud+schemaless (ex: 
> TestCloudManagedSchema) tests don't seem to demonstrate the same problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7669) Add SelectStream to Streaming API

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001680#comment-15001680
 ] 

ASF subversion and git services commented on SOLR-7669:
---

Commit 1713967 from dpg...@apache.org in branch 'dev/trunk'
[ https://svn.apache.org/r1713967 ]

SOLR-7669: Add SelectStream and Tuple Operations to the Streaming API and 
Streaming Expressions

> Add SelectStream to Streaming API
> -
>
> Key: SOLR-7669
> URL: https://issues.apache.org/jira/browse/SOLR-7669
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Reporter: Dennis Gove
>Assignee: Dennis Gove
>Priority: Minor
>  Labels: Streaming
> Attachments: SOLR-7669.patch, SOLR-7669.patch, SOLR-7669.patch, 
> SOLR-7669.patch, SOLR-7669.patch, SOLR-7669.patch
>
>
> Adds a new stream called SelectStream which can be used for two purpose.
>  1. Limit the set of fields included in an outgoing tuple to remove unwanted 
> fields
>  2. Provide aliases for fields. With this it acts as an alternative to the 
> CloudSolrStream's 'aliases' option.
>  For example, in a simple case
> {code}
> select(
>   id, 
>   fieldA_i as fieldA, 
>   fieldB_s as fieldB,
>   search(collection1, q="*:*", fl="id,fieldA_i,fieldB_s", sort="fieldA_i asc, 
> fieldB_s asc, id asc")
> )
> {code}
> This can also be used as part of complex expressions to help keep track of 
> what is being worked on. This is particularly useful when merging/joining 
> multiple collections which share field names. For example, the following 
> results in a set of tuples including only the fields id, left.ident, and 
> right.ident even though the total set of fields required to perform the 
> search and join is much larger than just those three fields.
> {code}
> select(
>   id, left.ident, right.ident,
>   innerJoin(
> select(
>   id, join1_i as left.join1, join2_s as left.join2, ident_s as left.ident,
>   search(collection1, q="side_s:left", fl="id,join1_i,join2_s,ident_s", 
> sort="join1_i asc, join2_s asc, id asc")
> ),
> select(
>   join3_i as right.join1, join2_s as right.join2, ident_s as right.ident,
>   search(collection1, q="side_s:right", fl="join3_i,join2_s,ident_s", 
> sort="join3_i asc, join2_s asc"),
> ),
> on="left.join1=right.join1, left.join2=right.join2"
>   )
> )
> {code}
> This depends on SOLR-7584.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8281) Add RollupMergeStream to Streaming API

[Joel Bernstein (JIRA)]

Joel Bernstein created SOLR-8281:


 Summary: Add RollupMergeStream to Streaming API
 Key: SOLR-8281
 URL: https://issues.apache.org/jira/browse/SOLR-8281
 Project: Solr
  Issue Type: Bug
Reporter: Joel Bernstein


The RollupMergeStream merges the aggregate results emitted by the RollupStream 
on *worker* nodes.

This is designed to be used in conjunction with the HashJoinStream to perform 
rollup Aggregations on the joined Tuples. The HashJoinStream will require the 
tuples to be partitioned on the Join keys. To avoid needing to repartition on 
the *group by* fields for the RollupStream, we can perform a merge of the 
rolled up Tuples coming from the workers.

The construct would like this:
{code}
mergeRollup (...
  parallel (...
hashJoin (
 search(...),
 search(...),
 on="fieldA" 
 )
 )
)
{code}


 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-6874:
--
Attachment: unicode-ws-tokenizer.patch

Here is my patch with the UnicodeWhitespaceTokenizer, small & elegant.

I have not added tests, I think we can take those from [~sar...@syr.edu]. I did 
not check performance, too. It is time for bed now.

To regenerate use {{ant regenerate}} or alternatively {{ant unicode-data}}.

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-8280) TestCloudSchemaless fails weirdly if you try to use SolrCoreAware sim factory: SchemaSimilarityFactory

[Hoss Man (JIRA)]

Hoss Man created SOLR-8280:
--

 Summary: TestCloudSchemaless fails weirdly if you try to use 
SolrCoreAware sim factory: SchemaSimilarityFactory 
 Key: SOLR-8280
 URL: https://issues.apache.org/jira/browse/SOLR-8280
 Project: Solr
  Issue Type: Bug
Affects Versions: Trunk
Reporter: Hoss Man


Something about the code path(s) involved in TestCloudSchemaless doesn't play 
nicely with a SimilarityFactory that is SolrCoreAware -- notably: 
SchemaSimilarityFactory.

I discovered this while trying to implement SOLR-8271, but it can be reproduced 
trivially by modifying the schema-add-schema-fields-update-processor.xml file 
used by TestCloudSchemaless to refer to SchemaSimilarityFactory explicitly.  
Other cloud tests (such as CollectionReloadTest) or cloud+schemaless (ex: 
TestCloudManagedSchema) tests don't seem to demonstrate the same problem.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8281) Add RollupMergeStream to Streaming API

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-8281:
-
Description: 
The RollupMergeStream merges the aggregate results emitted by the RollupStream 
on *worker* nodes.

This is designed to be used in conjunction with the HashJoinStream to perform 
rollup Aggregations on the joined Tuples. The HashJoinStream will require the 
tuples to be partitioned on the Join keys. To avoid needing to repartition on 
the *group by* fields for the RollupStream, we can perform a merge of the 
rolled up Tuples coming from the workers.

The construct would like this:
{code}
mergeRollup (...
  parallel (...
rollup (...
hashJoin (
  search(...),
  search(...),
  on="fieldA" 
)
 )
 )
   )
{code}

The pseudo code above would push the *hashJoin* and *rollup* to the *worker* 
nodes. The emitted rolled up tuples would be merged by the mergeRollup.


 

  was:
The RollupMergeStream merges the aggregate results emitted by the RollupStream 
on *worker* nodes.

This is designed to be used in conjunction with the HashJoinStream to perform 
rollup Aggregations on the joined Tuples. The HashJoinStream will require the 
tuples to be partitioned on the Join keys. To avoid needing to repartition on 
the *group by* fields for the RollupStream, we can perform a merge of the 
rolled up Tuples coming from the workers.

The construct would like this:
{code}
mergeRollup (...
  parallel (...
hashJoin (
 search(...),
 search(...),
 on="fieldA" 
 )
 )
)
{code}


 


> Add RollupMergeStream to Streaming API
> --
>
> Key: SOLR-8281
> URL: https://issues.apache.org/jira/browse/SOLR-8281
> Project: Solr
>  Issue Type: Bug
>Reporter: Joel Bernstein
>
> The RollupMergeStream merges the aggregate results emitted by the 
> RollupStream on *worker* nodes.
> This is designed to be used in conjunction with the HashJoinStream to perform 
> rollup Aggregations on the joined Tuples. The HashJoinStream will require the 
> tuples to be partitioned on the Join keys. To avoid needing to repartition on 
> the *group by* fields for the RollupStream, we can perform a merge of the 
> rolled up Tuples coming from the workers.
> The construct would like this:
> {code}
> mergeRollup (...
>   parallel (...
> rollup (...
> hashJoin (
>   search(...),
>   search(...),
>   on="fieldA" 
> )
>  )
>  )
>)
> {code}
> The pseudo code above would push the *hashJoin* and *rollup* to the *worker* 
> nodes. The emitted rolled up tuples would be merged by the mergeRollup.
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-8271) use SchemaSimilarityFactory as default when no explicit (top level) sim is configured

[Hoss Man (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-8271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Hoss Man updated SOLR-8271:
---
Attachment: SOLR-8271.patch

Initial simple patch, currently causes failures in TestCloudSchemaless & 
ChangedSchemaMergeTest.

These are the same failures I noted in the early attempts at SOLR-8057.  
Earlier today I thought that was because I was being silly in that old patch 
and needed to use the SolrResourceLoader to create the sin factory so 
SolrCoreAware.inform would be called appropriately in all situations -- which I 
do in this patch.  But the failures persist.  Digging into it I realized the 
same problem could easily be reproduced via configs -- so this issue is 
currently bloced until we can get to the bottom of SOLR-8280.


> use SchemaSimilarityFactory as default when no explicit (top level) sim is 
> configured
> -
>
> Key: SOLR-8271
> URL: https://issues.apache.org/jira/browse/SOLR-8271
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
> Attachments: SOLR-8271.patch
>
>
> Idea spun out of SOLR-8057...
> bq. As far as i can tell, the chief reason SchemaSimilarityFactory wasn't 
> made the implicit default in IndexSchema when it was introduced is because of 
> how it differed/differs from DefaultSimilarity/ClassicSimilarity with respect 
> to multi-clause queries – see SchemaSimilarityFactory's class javadoc notes 
> relating to {{queryNorm}} and {{coord}}. Users were expected to think about 
> this trade off when making a concious choice to switch from 
> DefaultSimilarity/ClassicSimilarity to SchemaSimilarityFactory. But (again, 
> AFAICT) these discrepencies don't exist between SchemaSimilarityFactory's 
> PerFieldSimilarityWrapper and BM25Similiarity.
> So assuming luceneMatchVersion >= 6.0, and BM25 is implicit default, we 
> should be able to safely switch to using SchemaSimilarityFactory as our 
> default (which internally uses BM25 for fieldTypes that don't override) and 
> make it much easier for people to declare fieldType overrides for the 
> similarity (just edit the fieldType, w/o also needing to explicitly declare 
> SchemaSimilarityFactory)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001419#comment-15001419
 ] 

Uwe Schindler edited comment on LUCENE-6874 at 11/12/15 12:42 AM:
--

Here is my patch with the UnicodeWhitespaceTokenizer, small & elegant.

I have not added tests, I think we can take those from [~steve_rowe]. I did not 
check performance, too. It is time for bed now.

To regenerate use {{ant regenerate}} or alternatively {{ant unicode-data}}.


was (Author: thetaphi):
Here is my patch with the UnicodeWhitespaceTokenizer, small & elegant.

I have not added tests, I think we can take those from [~sar...@syr.edu]. I did 
not check performance, too. It is time for bed now.

To regenerate use {{ant regenerate}} or alternatively {{ant unicode-data}}.

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-7669) Add SelectStream to Streaming API

[Dennis Gove (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dennis Gove reassigned SOLR-7669:
-

Assignee: Dennis Gove

> Add SelectStream to Streaming API
> -
>
> Key: SOLR-7669
> URL: https://issues.apache.org/jira/browse/SOLR-7669
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Reporter: Dennis Gove
>Assignee: Dennis Gove
>Priority: Minor
>  Labels: Streaming
> Attachments: SOLR-7669.patch, SOLR-7669.patch, SOLR-7669.patch, 
> SOLR-7669.patch, SOLR-7669.patch
>
>
> Adds a new stream called SelectStream which can be used for two purpose.
>  1. Limit the set of fields included in an outgoing tuple to remove unwanted 
> fields
>  2. Provide aliases for fields. With this it acts as an alternative to the 
> CloudSolrStream's 'aliases' option.
>  For example, in a simple case
> {code}
> select(
>   id, 
>   fieldA_i as fieldA, 
>   fieldB_s as fieldB,
>   search(collection1, q="*:*", fl="id,fieldA_i,fieldB_s", sort="fieldA_i asc, 
> fieldB_s asc, id asc")
> )
> {code}
> This can also be used as part of complex expressions to help keep track of 
> what is being worked on. This is particularly useful when merging/joining 
> multiple collections which share field names. For example, the following 
> results in a set of tuples including only the fields id, left.ident, and 
> right.ident even though the total set of fields required to perform the 
> search and join is much larger than just those three fields.
> {code}
> select(
>   id, left.ident, right.ident,
>   innerJoin(
> select(
>   id, join1_i as left.join1, join2_s as left.join2, ident_s as left.ident,
>   search(collection1, q="side_s:left", fl="id,join1_i,join2_s,ident_s", 
> sort="join1_i asc, join2_s asc, id asc")
> ),
> select(
>   join3_i as right.join1, join2_s as right.join2, ident_s as right.ident,
>   search(collection1, q="side_s:right", fl="join3_i,join2_s,ident_s", 
> sort="join3_i asc, join2_s asc"),
> ),
> on="left.join1=right.join1, left.join2=right.join2"
>   )
> )
> {code}
> This depends on SOLR-7584.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-trunk-Linux (64bit/jdk1.9.0-ea-b90) - Build # 14876 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-trunk-Linux/14876/
Java: 64bit/jdk1.9.0-ea-b90 -XX:+UseCompressedOops -XX:+UseSerialGC

3 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.SaslZkACLProviderTest

Error Message:
5 threads leaked from SUITE scope at 
org.apache.solr.cloud.SaslZkACLProviderTest: 1) Thread[id=3243, 
name=groupCache.data, state=TIMED_WAITING, group=TGRP-SaslZkACLProviderTest]
 at sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:218) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2103)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1136)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:853)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1083)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632) 
at java.lang.Thread.run(Thread.java:747)2) Thread[id=3242, 
name=kdcReplayCache.data, state=TIMED_WAITING, 
group=TGRP-SaslZkACLProviderTest] at sun.misc.Unsafe.park(Native 
Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:218) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2103)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1136)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:853)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1083)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632) 
at java.lang.Thread.run(Thread.java:747)3) Thread[id=3244, 
name=ou=system.data, state=TIMED_WAITING, group=TGRP-SaslZkACLProviderTest] 
at sun.misc.Unsafe.park(Native Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:218) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2103)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1136)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:853)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1083)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632) 
at java.lang.Thread.run(Thread.java:747)4) Thread[id=3241, 
name=apacheds, state=WAITING, group=TGRP-SaslZkACLProviderTest] at 
java.lang.Object.wait(Native Method) at 
java.lang.Object.wait(Object.java:516) at 
java.util.TimerThread.mainLoop(Timer.java:526) at 
java.util.TimerThread.run(Timer.java:505)5) Thread[id=3245, 
name=changePwdReplayCache.data, state=TIMED_WAITING, 
group=TGRP-SaslZkACLProviderTest] at sun.misc.Unsafe.park(Native 
Method) at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:218) 
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2103)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1136)
 at 
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:853)
 at 
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1083)   
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1143) 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:632) 
at java.lang.Thread.run(Thread.java:747)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 5 threads leaked from SUITE 
scope at org.apache.solr.cloud.SaslZkACLProviderTest: 
   1) Thread[id=3243, name=groupCache.data, state=TIMED_WAITING, 
group=TGRP-SaslZkACLProviderTest]
at sun.misc.Unsafe.park(Native Method)
at 
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:218)
at 
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2103)
at 

[jira] [Updated] (SOLR-7669) Add SelectStream to Streaming API

[Dennis Gove (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dennis Gove updated SOLR-7669:
--
Attachment: SOLR-7669.patch

Fixes for pre-commit failures. Add documentation on the operations.

> Add SelectStream to Streaming API
> -
>
> Key: SOLR-7669
> URL: https://issues.apache.org/jira/browse/SOLR-7669
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrJ
>Reporter: Dennis Gove
>Assignee: Dennis Gove
>Priority: Minor
>  Labels: Streaming
> Attachments: SOLR-7669.patch, SOLR-7669.patch, SOLR-7669.patch, 
> SOLR-7669.patch, SOLR-7669.patch, SOLR-7669.patch
>
>
> Adds a new stream called SelectStream which can be used for two purpose.
>  1. Limit the set of fields included in an outgoing tuple to remove unwanted 
> fields
>  2. Provide aliases for fields. With this it acts as an alternative to the 
> CloudSolrStream's 'aliases' option.
>  For example, in a simple case
> {code}
> select(
>   id, 
>   fieldA_i as fieldA, 
>   fieldB_s as fieldB,
>   search(collection1, q="*:*", fl="id,fieldA_i,fieldB_s", sort="fieldA_i asc, 
> fieldB_s asc, id asc")
> )
> {code}
> This can also be used as part of complex expressions to help keep track of 
> what is being worked on. This is particularly useful when merging/joining 
> multiple collections which share field names. For example, the following 
> results in a set of tuples including only the fields id, left.ident, and 
> right.ident even though the total set of fields required to perform the 
> search and join is much larger than just those three fields.
> {code}
> select(
>   id, left.ident, right.ident,
>   innerJoin(
> select(
>   id, join1_i as left.join1, join2_s as left.join2, ident_s as left.ident,
>   search(collection1, q="side_s:left", fl="id,join1_i,join2_s,ident_s", 
> sort="join1_i asc, join2_s asc, id asc")
> ),
> select(
>   join3_i as right.join1, join2_s as right.join2, ident_s as right.ident,
>   search(collection1, q="side_s:right", fl="join3_i,join2_s,ident_s", 
> sort="join3_i asc, join2_s asc"),
> ),
> on="left.join1=right.join1, left.join2=right.join2"
>   )
> )
> {code}
> This depends on SOLR-7584.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[Uwe Schindler (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Uwe Schindler updated LUCENE-6874:
--
Attachment: unicode-ws-tokenizer.patch

Minor changes

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch, 
> unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-5.x-Solaris (multiarch/jdk1.7.0) - Build # 182 - Still Failing!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-5.x-Solaris/182/
Java: multiarch/jdk1.7.0 -d64 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

1 tests failed.
FAILED:  org.apache.solr.cloud.hdfs.StressHdfsTest.test

Error Message:
Could not load collection from ZK:delete_data_dir

Stack Trace:
org.apache.solr.common.SolrException: Could not load collection from 
ZK:delete_data_dir
at 
__randomizedtesting.SeedInfo.seed([AB564B1476D193B:82E15B6BE99174C3]:0)
at 
org.apache.solr.common.cloud.ZkStateReader.getCollectionLive(ZkStateReader.java:1022)
at 
org.apache.solr.common.cloud.ZkStateReader$LazyCollectionRef.get(ZkStateReader.java:555)
at 
org.apache.solr.common.cloud.ClusterState.getCollectionOrNull(ClusterState.java:193)
at 
org.apache.solr.common.cloud.ClusterState.hasCollection(ClusterState.java:123)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.getCollectionNames(CloudSolrClient.java:1110)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:833)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:806)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:150)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:483)
at org.apache.solr.client.solrj.SolrClient.commit(SolrClient.java:464)
at 
org.apache.solr.cloud.hdfs.StressHdfsTest.createAndDeleteCollection(StressHdfsTest.java:197)
at 
org.apache.solr.cloud.hdfs.StressHdfsTest.test(StressHdfsTest.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1660)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:866)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:902)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:916)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:963)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:938)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:50)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:49)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:65)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:48)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:875)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:777)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:811)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:822)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:46)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:42)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[jira] [Comment Edited] (LUCENE-6874) WhitespaceTokenizer should tokenize on NBSP

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15001688#comment-15001688
 ] 

David Smiley edited comment on LUCENE-6874 at 11/12/15 4:47 AM:


+1 I like it Uwe; nice job.  Automating the generation of the unicode code 
points is good, and may come in handy if we want other whitespace 
rules/definitions.


was (Author: dsmiley):
+1 I like it Uwe; nice job.  Automating the generation of the unicode code 
points is good, and may come in handy if we want other unicode rules.

> WhitespaceTokenizer should tokenize on NBSP
> ---
>
> Key: LUCENE-6874
> URL: https://issues.apache.org/jira/browse/LUCENE-6874
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: David Smiley
>Priority: Minor
> Attachments: LUCENE-6874-jflex.patch, LUCENE-6874.patch, 
> LUCENE_6874_jflex.patch, icu-datasucker.patch, unicode-ws-tokenizer.patch
>
>
> WhitespaceTokenizer uses [Character.isWhitespace 
> |http://docs.oracle.com/javase/8/docs/api/java/lang/Character.html#isWhitespace-int-]
>  to decide what is whitespace.  Here's a pertinent excerpt:
> bq. It is a Unicode space character (SPACE_SEPARATOR, LINE_SEPARATOR, or 
> PARAGRAPH_SEPARATOR) but is not also a non-breaking space ('\u00A0', 
> '\u2007', '\u202F')
> Perhaps Character.isWhitespace should have been called 
> isLineBreakableWhitespace?
> I think WhitespaceTokenizer should tokenize on this.  I am aware it's easy to 
> work around but why leave this trap in by default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Welcome Areek Zillur to the Lucene / Solr PMC

[david.w.smi...@gmail.com]

Welcome Areek!

On Wed, Nov 11, 2015 at 3:49 PM Simon Willnauer 
wrote:

> I'm pleased to announce that Areek has accepted the PMC's invitation to
> join.
>
> Welcome Areek!
>
>
> Simon
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com