[jira] [Commented] (SOLR-9529) Dates Dynamic Field Inconsistently Defined in Schemas

[Alexandre Rafalovitch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500327#comment-15500327
 ] 

Alexandre Rafalovitch commented on SOLR-9529:
-

We also have inconsistency with docValues, we have field types present in one 
but not another, etc.

The question is also whether we really need to ship most of the definitions 
with each example? This causes both drift over type and confusion for users.

One alternative is to have absolutely minimal examples (plus one kitchen sink) 
and then have a mechanism to add those additional field definitions as needed 
via REST API. Perhaps with using Admin UI and some sort of templates.

> Dates Dynamic Field Inconsistently Defined in Schemas
> -
>
> Key: SOLR-9529
> URL: https://issues.apache.org/jira/browse/SOLR-9529
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Trey Grainger
>Priority: Minor
>
> There is a nice convention across all of the schemas that ship with Solr to 
> include field types for single valued fields (i.e. "string" -> "*_s", 
> "boolean" -> "*_b") and separate field types for multivalued fields (i.e. 
> "strings" -> "*_ss", "booleans" -> "*_bs"). Those definitions all follow the 
> pattern (using "string" as an example):
> 
>  multiValued="true"/>
> 
> 
> For some reason, however, the "date" field type doesn't follow this pattern, 
> and is instead defined (inconsistently) as follows:
>  precisionStep="0"/>
>  multiValued="true" precisionStep="0"/>
> 
>  stored="true"/>
> Note specifically that the "*_dts" field should instead be referencing the 
> "dates" type and not the "date" type, and that subsequently the 
> multiValued="true" setting would become unnecessary on the "*_dts" 
> dynamicField definition.
> I'll get a patch posted for this. Note that nothing is functionally broken, 
> it's just inconsistent and could be confusing for someone looking through the 
> schema or seeing their multivalued dates getting indexed into the field type 
> defined for single valued dates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-9515) Update to Hadoop 3

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-9515:
--
Attachment: SOLR-9515.patch

This is almost everything. Main issue left is that a kerberos test is not 
working with the minikdc for some reason.

> Update to Hadoop 3
> --
>
> Key: SOLR-9515
> URL: https://issues.apache.org/jira/browse/SOLR-9515
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
> Attachments: SOLR-9515.patch
>
>
> Hadoop 3 is not out yet, but I'd like to iron out the upgrade to be prepared. 
> I'll start up a dev branch.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9529) Dates Dynamic Field Inconsistently Defined in Schemas

[Trey Grainger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500294#comment-15500294
 ] 

Trey Grainger commented on SOLR-9529:
-

Hmm... things were more inconsistent than I thought. There were two fundamental 
kinds of inconsistencies:
1) Inconsistencies within a single schema.
--This is what I described in the issue description regarding "*_dts" being 
handled incorrectly. I submitted a pull request to fix this in the three places 
we actually define both singular and plural field types: 
solr/example/files/conf/managed-schema
solr/server/solr/configsets/basic_configs/conf/managed-schema
solr/server/solr/configsets/data_driven_schema_configs/conf/managed-schema

2) Inconsistencies across different schemas
While the three schemas listed above all separate out single valued and 
multiValued dynamic fields into different singular and plural field types, 
every other schema that ships with Solr only defines a single field type 
(string, boolean, etc.) and uses the dynamic field definition to determine 
whether the dynamic field should be single or multivalued. This works fine, of 
course, but is just inconsistent depending upon which schema file you actually 
end up using. 

Interestingly, the tech products example 
(solr/server/solr/configsets/sample_techproducts_configs/conf/managed-schema), 
which sits at the same level as the basic_configs and the 
data_driven_schema_configs, for some reason handles these definitions 
differently, only defining one field type for both single and multivalued 
fields (for all types). The following places do the same thing:
 
solr/core/src/test-files/solr/collection1/conf/schema-distrib-interval-faceting.xml
 solr/core/src/test-files/solr/collection1/conf/schema-docValuesFaceting.xml
 solr/core/src/test-files/solr/collection1/conf/schema-docValuesJoin.xml
 
solr/core/src/test-files/solr/collection1/conf/schema-non-stored-docvalues.xml
 solr/core/src/test-files/solr/collection1/conf/schema_latest.xml
 solr/example/example-DIH/solr/db/conf/managed-schema
 solr/example/example-DIH/solr/mail/conf/managed-schema
 solr/example/example-DIH/solr/rss/conf/managed-schema
 solr/example/example-DIH/solr/solr/conf/managed-schema
 solr/example/example-DIH/solr/tika/conf/managed-schema

So while my pull request fixes #1 so that all schemas are consistent with 
themselves, we still have inconsistency across the various schemas that ship 
with Solr in terms of what we name the field types for multivalued dynamic 
fields. If we are going to make these consistent, which way should we go - have 
a single field type for all single and multivalued fields (and define 
multivalued=true on the dynamic field definition instead), or separate out 
plural versions of the field type (booleans, strings, etc.) for multivalued 
fields?

> Dates Dynamic Field Inconsistently Defined in Schemas
> -
>
> Key: SOLR-9529
> URL: https://issues.apache.org/jira/browse/SOLR-9529
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Trey Grainger
>Priority: Minor
>
> There is a nice convention across all of the schemas that ship with Solr to 
> include field types for single valued fields (i.e. "string" -> "*_s", 
> "boolean" -> "*_b") and separate field types for multivalued fields (i.e. 
> "strings" -> "*_ss", "booleans" -> "*_bs"). Those definitions all follow the 
> pattern (using "string" as an example):
> 
>  multiValued="true"/>
> 
> 
> For some reason, however, the "date" field type doesn't follow this pattern, 
> and is instead defined (inconsistently) as follows:
>  precisionStep="0"/>
>  multiValued="true" precisionStep="0"/>
> 
>  stored="true"/>
> Note specifically that the "*_dts" field should instead be referencing the 
> "dates" type and not the "date" type, and that subsequently the 
> multiValued="true" setting would become unnecessary on the "*_dts" 
> dynamicField definition.
> I'll get a patch posted for this. Note that nothing is functionally broken, 
> it's just inconsistent and could be confusing for someone looking through the 
> schema or seeing their multivalued dates getting indexed into the field type 
> defined for single valued dates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 852 - Still Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/852/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 1) Thread[id=7442, 
name=OverseerHdfsCoreFailoverThread-96611439029059596-127.0.0.1:63743_solr-n_02,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at 
java.lang.Thread.sleep(Native Method) at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
 at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates: 
   1) Thread[id=7442, 
name=OverseerHdfsCoreFailoverThread-96611439029059596-127.0.0.1:63743_solr-n_02,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([BDF50524D5B09223]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates

Error Message:
There are still zombie threads that couldn't be terminated:1) 
Thread[id=7442, 
name=OverseerHdfsCoreFailoverThread-96611439029059596-127.0.0.1:63743_solr-n_02,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.] at 
java.lang.Thread.sleep(Native Method) at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
 at java.lang.Thread.run(Thread.java:745)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=7442, 
name=OverseerHdfsCoreFailoverThread-96611439029059596-127.0.0.1:63743_solr-n_02,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(Native Method)
at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
at java.lang.Thread.run(Thread.java:745)
at __randomizedtesting.SeedInfo.seed([BDF50524D5B09223]:0)




Build Log:
[...truncated 11167 lines...]
   [junit4] Suite: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
   [junit4]   2> Creating dataDir: 
/export/home/jenkins/workspace/Lucene-Solr-master-Solaris/solr/build/solr-core/test/J1/temp/solr.cloud.TestStressCloudBlindAtomicUpdates_BDF50524D5B09223-001/init-core-data-001
   [junit4]   2> 720584 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (false) via: 
@org.apache.solr.SolrTestCaseJ4$SuppressSSL(bugUrl=SSL overhead seems to cause 
OutOfMemory when stress testing)
   [junit4]   2> 720584 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 720584 INFO  (Thread-1738) [] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 720585 INFO  (Thread-1738) [] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 720685 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.ZkTestServer start zk server on port:45087
   [junit4]   2> 720685 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 720685 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 720688 INFO  (zkCallback-1323-thread-1) [] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@6d973335 
name:ZooKeeperConnection Watcher:127.0.0.1:45087 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 720688 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 720688 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 720688 INFO  
(SUITE-TestStressCloudBlindAtomicUpdates-seed#[BDF50524D5B09223]-worker) [] 
o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 720695 INFO  (jetty-launcher-1322-thread-1) [] 
o.e.j.s.Server jetty-9.3.8.v20160314
   [junit4]   2> 720696 INFO  (jetty-launcher-1322-thread-4) [] 
o.e.j.s.Server 

[jira] [Commented] (SOLR-9529) Dates Dynamic Field Inconsistently Defined in Schemas

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15500265#comment-15500265
 ] 

ASF GitHub Bot commented on SOLR-9529:
--

GitHub user treygrainger opened a pull request:

https://github.com/apache/lucene-solr/pull/80

SOLR-9529 - Make multivalued dates dynamic field use dates field type



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/treygrainger/lucene-solr SOLR-9529

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/80.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #80


commit 5428f19bdc72b2d2e55da465e99c4419449dca97
Author: Trey Grainger 
Date:   2016-09-18T04:53:34Z

SOLR-9529 - Make multivalued dates dynamic field use dates field type




> Dates Dynamic Field Inconsistently Defined in Schemas
> -
>
> Key: SOLR-9529
> URL: https://issues.apache.org/jira/browse/SOLR-9529
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Trey Grainger
>Priority: Minor
>
> There is a nice convention across all of the schemas that ship with Solr to 
> include field types for single valued fields (i.e. "string" -> "*_s", 
> "boolean" -> "*_b") and separate field types for multivalued fields (i.e. 
> "strings" -> "*_ss", "booleans" -> "*_bs"). Those definitions all follow the 
> pattern (using "string" as an example):
> 
>  multiValued="true"/>
> 
> 
> For some reason, however, the "date" field type doesn't follow this pattern, 
> and is instead defined (inconsistently) as follows:
>  precisionStep="0"/>
>  multiValued="true" precisionStep="0"/>
> 
>  stored="true"/>
> Note specifically that the "*_dts" field should instead be referencing the 
> "dates" type and not the "date" type, and that subsequently the 
> multiValued="true" setting would become unnecessary on the "*_dts" 
> dynamicField definition.
> I'll get a patch posted for this. Note that nothing is functionally broken, 
> it's just inconsistent and could be confusing for someone looking through the 
> schema or seeing their multivalued dates getting indexed into the field type 
> defined for single valued dates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #80: SOLR-9529 - Make multivalued dates dynamic fie...

[treygrainger]

GitHub user treygrainger opened a pull request:

https://github.com/apache/lucene-solr/pull/80

SOLR-9529 - Make multivalued dates dynamic field use dates field type



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/treygrainger/lucene-solr SOLR-9529

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/80.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #80


commit 5428f19bdc72b2d2e55da465e99c4419449dca97
Author: Trey Grainger 
Date:   2016-09-18T04:53:34Z

SOLR-9529 - Make multivalued dates dynamic field use dates field type




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9529) Dates Dynamic Field Inconsistently Defined in Schemas

[Trey Grainger (JIRA)]

Trey Grainger created SOLR-9529:
---

 Summary: Dates Dynamic Field Inconsistently Defined in Schemas
 Key: SOLR-9529
 URL: https://issues.apache.org/jira/browse/SOLR-9529
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Trey Grainger
Priority: Minor


There is a nice convention across all of the schemas that ship with Solr to 
include field types for single valued fields (i.e. "string" -> "*_s", "boolean" 
-> "*_b") and separate field types for multivalued fields (i.e. "strings" -> 
"*_ss", "booleans" -> "*_bs"). Those definitions all follow the pattern (using 
"string" as an example):






For some reason, however, the "date" field type doesn't follow this pattern, 
and is instead defined (inconsistently) as follows:





Note specifically that the "*_dts" field should instead be referencing the 
"dates" type and not the "date" type, and that subsequently the 
multiValued="true" setting would become unnecessary on the "*_dts" dynamicField 
definition.

I'll get a patch posted for this. Note that nothing is functionally broken, 
it's just inconsistent and could be confusing for someone looking through the 
schema or seeing their multivalued dates getting indexed into the field type 
defined for single valued dates.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-6.2 - Build # 6 - Still unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-6.2/6/

5 tests failed.
FAILED:  org.apache.solr.cloud.FullSolrCloudDistribCmdsTest.test

Error Message:
Captured an uncaught exception in thread: Thread[id=22586, name=Thread-17491, 
state=RUNNABLE, group=TGRP-FullSolrCloudDistribCmdsTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=22586, name=Thread-17491, state=RUNNABLE, 
group=TGRP-FullSolrCloudDistribCmdsTest]
Caused by: java.lang.RuntimeException: 
org.apache.solr.client.solrj.SolrServerException: Timeout occured while waiting 
response from server at: http://127.0.0.1:42385/ls/pc/collection1
at __randomizedtesting.SeedInfo.seed([5ECF1679EBCA22A3]:0)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:644)
Caused by: org.apache.solr.client.solrj.SolrServerException: Timeout occured 
while waiting response from server at: http://127.0.0.1:42385/ls/pc/collection1
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:603)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:261)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:250)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:149)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:166)
at 
org.apache.solr.cloud.FullSolrCloudDistribCmdsTest$1IndexThread.run(FullSolrCloudDistribCmdsTest.java:642)
Caused by: java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:160)
at 
org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:84)
at 
org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:273)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:140)
at 
org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
at 
org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
at 
org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:283)
at 
org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:251)
at 
org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:197)
at 
org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
at 
org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
at 
org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:685)
at 
org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:487)
at 
org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at 
org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:497)
... 5 more


FAILED:  org.apache.solr.cloud.TestCryptoKeys.test

Error Message:
Could not find collection:.system

Stack Trace:
java.lang.AssertionError: Could not find collection:.system
at 
__randomizedtesting.SeedInfo.seed([5ECF1679EBCA22A3:D69B29A345364F5B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at org.apache.solr.cloud.TestCryptoKeys.test(TestCryptoKeys.java:117)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 

[jira] [Updated] (SOLR-9480) Graph Traversal for Significantly Related Terms (Semantic Knowledge Graph)

[Trey Grainger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Trey Grainger updated SOLR-9480:

Attachment: SOLR-9480.patch

Initial patch to get the ball rolling here. Feature should now work as 
described in reference links in the description. Only real changes are an 
update from Solr 5.1.0 to master, and cleanup of most of the precommit issues.

Still plenty of work to do, particularly in reworking some of the 
multi-threading code to follow Solr conventions, reducing the number of files 
for helper classes, and eventually getting this working correctly in 
distributed mode (was originally built for use cases involving a single Solr 
core as a "representative model"). Would also be good to make a getting started 
tutorial with example data so its easier get started with the feature and do 
something interesting out of the box.

Will continue working on those items as I'm able. Feedback welcome.

> Graph Traversal for Significantly Related Terms (Semantic Knowledge Graph)
> --
>
> Key: SOLR-9480
> URL: https://issues.apache.org/jira/browse/SOLR-9480
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Trey Grainger
> Attachments: SOLR-9480.patch
>
>
> This issue is to track the contribution of the Semantic Knowledge Graph Solr 
> Plugin (request handler), which exposes a graph-like interface for 
> discovering and traversing significant relationships between entities within 
> an inverted index.
> This data model has been described in the following research paper: [The 
> Semantic Knowledge Graph: A compact, auto-generated model for real-time 
> traversal and ranking of any relationship within a 
> domain|https://arxiv.org/abs/1609.00464], as well as in presentations I gave 
> in October 2015 at [Lucene/Solr 
> Revolution|http://www.slideshare.net/treygrainger/leveraging-lucenesolr-as-a-knowledge-graph-and-intent-engine]
>  and November 2015 at the [Bay Area Search 
> Meetup|http://www.treygrainger.com/posts/presentations/searching-on-intent-knowledge-graphs-personalization-and-contextual-disambiguation/].
> The source code for this project is currently available at 
> [https://github.com/careerbuilder/semantic-knowledge-graph], and the folks at 
> CareerBuilder (where this was built) have given me the go-ahead to now 
> contribute this back to the Apache Solr Project, as well.
> Check out the Github repository, research paper, or presentations for a more 
> detailed description of this contribution. Initial patch coming soon.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-9528) Make _docid_ (lucene id) a pseudo field

[Alexandre Rafalovitch (JIRA)]

Alexandre Rafalovitch created SOLR-9528:
---

 Summary: Make _docid_ (lucene id) a pseudo field
 Key: SOLR-9528
 URL: https://issues.apache.org/jira/browse/SOLR-9528
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: master (7.0)
Reporter: Alexandre Rafalovitch
Priority: Minor


Lucene document id is a transitory id that cannot be relied on as it can change 
on document updates, etc.

However, there are circumstances where it could be useful to use it in a 
search. The primarily use is a debugging where some error messages provide only 
lucene document id as the reference. For example:

{noformat}
child query must only match non-parent docs, but parent docID=38200 matched 
childScorer=class org.apache.lucene.search.DisjunctionSumScorer
{noformat}

We already expose the lucene id with \[docid] transformer with \_docid_ sorting.

On the email list, [~yo...@apache.org] proposed that _docid_ should be a 
legitimate pseudo-field, which would make it returnable, usable in function 
queries, etc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-master-Linux (64bit/jdk-9-ea+134) - Build # 17839 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/17839/
Java: 64bit/jdk-9-ea+134 -XX:+UseCompressedOops -XX:+UseSerialGC

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.search.stats.TestDistribIDF

Error Message:
1 thread leaked from SUITE scope at 
org.apache.solr.search.stats.TestDistribIDF: 1) Thread[id=94429, 
name=OverseerHdfsCoreFailoverThread-96609969231298569-127.0.0.1:34549_solr-n_02,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.] at 
java.lang.Thread.sleep(java.base@9-ea/Native Method) at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: 1 thread leaked from SUITE 
scope at org.apache.solr.search.stats.TestDistribIDF: 
   1) Thread[id=94429, 
name=OverseerHdfsCoreFailoverThread-96609969231298569-127.0.0.1:34549_solr-n_02,
 state=TIMED_WAITING, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(java.base@9-ea/Native Method)
at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
at __randomizedtesting.SeedInfo.seed([E56F3B34DE9261B7]:0)


FAILED:  junit.framework.TestSuite.org.apache.solr.search.stats.TestDistribIDF

Error Message:
There are still zombie threads that couldn't be terminated:1) 
Thread[id=94429, 
name=OverseerHdfsCoreFailoverThread-96609969231298569-127.0.0.1:34549_solr-n_02,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.] at 
java.lang.Thread.sleep(java.base@9-ea/Native Method) at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
 at java.lang.Thread.run(java.base@9-ea/Thread.java:843)

Stack Trace:
com.carrotsearch.randomizedtesting.ThreadLeakError: There are still zombie 
threads that couldn't be terminated:
   1) Thread[id=94429, 
name=OverseerHdfsCoreFailoverThread-96609969231298569-127.0.0.1:34549_solr-n_02,
 state=RUNNABLE, group=Overseer Hdfs SolrCore Failover Thread.]
at java.lang.Thread.sleep(java.base@9-ea/Native Method)
at 
org.apache.solr.cloud.OverseerAutoReplicaFailoverThread.run(OverseerAutoReplicaFailoverThread.java:137)
at java.lang.Thread.run(java.base@9-ea/Thread.java:843)
at __randomizedtesting.SeedInfo.seed([E56F3B34DE9261B7]:0)




Build Log:
[...truncated 12508 lines...]
   [junit4] Suite: org.apache.solr.search.stats.TestDistribIDF
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J1/temp/solr.search.stats.TestDistribIDF_E56F3B34DE9261B7-001/init-core-data-001
   [junit4]   2> 2045597 INFO  
(SUITE-TestDistribIDF-seed#[E56F3B34DE9261B7]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (false) and clientAuth (true) via: 
@org.apache.solr.util.RandomizeSSL(reason="", ssl=0.0/0.0, value=0.0/0.0, 
clientAuth=0.0/0.0)
   [junit4]   2> 2045601 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.SolrTestCaseJ4 ###Starting testSimpleQuery
   [junit4]   2> 2045614 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.ZkTestServer STARTING ZK TEST SERVER
   [junit4]   2> 2045614 INFO  (Thread-2460) [] o.a.s.c.ZkTestServer client 
port:0.0.0.0/0.0.0.0:0
   [junit4]   2> 2045614 INFO  (Thread-2460) [] o.a.s.c.ZkTestServer 
Starting server
   [junit4]   2> 2045714 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.ZkTestServer start zk server on port:44862
   [junit4]   2> 2045715 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.c.SolrZkClient Using default ZkCredentialsProvider
   [junit4]   2> 2045715 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.c.ConnectionManager Waiting for client to connect to ZooKeeper
   [junit4]   2> 2045719 INFO  (zkCallback-24697-thread-1) [] 
o.a.s.c.c.ConnectionManager Watcher 
org.apache.solr.common.cloud.ConnectionManager@37c351c4 
name:ZooKeeperConnection Watcher:127.0.0.1:44862 got event WatchedEvent 
state:SyncConnected type:None path:null path:null type:None
   [junit4]   2> 2045719 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.c.ConnectionManager Client is connected to ZooKeeper
   [junit4]   2> 2045719 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.c.SolrZkClient Using default ZkACLProvider
   [junit4]   2> 2045719 INFO  
(TEST-TestDistribIDF.testSimpleQuery-seed#[E56F3B34DE9261B7]) [] 
o.a.s.c.c.SolrZkClient makePath: /solr/solr.xml
   [junit4]   2> 2045721 INFO  (jetty-launcher-24696-thread-2) [] 
o.e.j.s.Server 

[JENKINS] Lucene-Solr-6.2-Windows (64bit/jdk1.8.0_102) - Build # 9 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.2-Windows/9/
Java: 64bit/jdk1.8.0_102 -XX:+UseCompressedOops -XX:+UseParallelGC

6 tests failed.
FAILED:  
org.apache.solr.cloud.CdcrBootstrapTest.testBootstrapWithContinousIndexingOnSourceCluster

Error Message:
Captured an uncaught exception in thread: Thread[id=15575, name=Thread-3574, 
state=RUNNABLE, group=TGRP-CdcrBootstrapTest]

Stack Trace:
com.carrotsearch.randomizedtesting.UncaughtExceptionError: Captured an uncaught 
exception in thread: Thread[id=15575, name=Thread-3574, state=RUNNABLE, 
group=TGRP-CdcrBootstrapTest]
at 
__randomizedtesting.SeedInfo.seed([33B5C5ACC3AFCB9A:E7F08EF524F97861]:0)
Caused by: java.lang.AssertionError: 1
at __randomizedtesting.SeedInfo.seed([33B5C5ACC3AFCB9A]:0)
at 
org.apache.solr.core.CachingDirectoryFactory.close(CachingDirectoryFactory.java:191)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1274)
at 
org.apache.solr.core.CoreContainer.registerCore(CoreContainer.java:701)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:941)
at 
org.apache.solr.handler.IndexFetcher.lambda$reloadCore$0(IndexFetcher.java:773)
at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.CdcrBootstrapTest

Error Message:
ObjectTracker found 4 object(s) that were not released!!! 
[MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, MockDirectoryWrapper]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 4 object(s) that were not 
released!!! [MockDirectoryWrapper, MockDirectoryWrapper, SolrCore, 
MockDirectoryWrapper]
at __randomizedtesting.SeedInfo.seed([33B5C5ACC3AFCB9A]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:258)
at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  junit.framework.TestSuite.org.apache.solr.schema.TestManagedSchemaAPI

Error Message:
ObjectTracker found 10 object(s) that were not released!!! 
[MockDirectoryWrapper, MockDirectoryWrapper, TransactionLog, TransactionLog, 
MDCAwareThreadPoolExecutor, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MDCAwareThreadPoolExecutor]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 10 object(s) that were not 
released!!! [MockDirectoryWrapper, MockDirectoryWrapper, TransactionLog, 
TransactionLog, MDCAwareThreadPoolExecutor, MockDirectoryWrapper, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MDCAwareThreadPoolExecutor]
at 

[JENKINS] Lucene-Solr-master-Windows (32bit/jdk1.8.0_102) - Build # 6123 - Still Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6123/
Java: 32bit/jdk1.8.0_102 -client -XX:+UseSerialGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail

Error Message:
expected:<200> but was:<404>

Stack Trace:
java.lang.AssertionError: expected:<200> but was:<404>
at 
__randomizedtesting.SeedInfo.seed([7ED9EC9E3B1A767A:1666D9B4EB806496]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.cancelDelegationToken(TestSolrCloudWithDelegationTokens.java:137)
at 
org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenCancelFail(TestSolrCloudWithDelegationTokens.java:282)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[JENKINS-EA] Lucene-Solr-6.2-Linux (64bit/jdk-9-ea+134) - Build # 28 - Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.2-Linux/28/
Java: 64bit/jdk-9-ea+134 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.TestCryptoKeys.test

Error Message:
Could not find collection:.system

Stack Trace:
java.lang.AssertionError: Could not find collection:.system
at 
__randomizedtesting.SeedInfo.seed([1D79292043F145B1:952D16FAED0D2849]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNotNull(Assert.java:526)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:154)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:139)
at 
org.apache.solr.cloud.AbstractDistribZkTestBase.waitForRecoveriesToFinish(AbstractDistribZkTestBase.java:134)
at 
org.apache.solr.cloud.AbstractFullDistribZkTestBase.waitForRecoveriesToFinish(AbstractFullDistribZkTestBase.java:856)
at org.apache.solr.cloud.TestCryptoKeys.test(TestCryptoKeys.java:117)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@9-ea/Native 
Method)
at 
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@9-ea/NativeMethodAccessorImpl.java:62)
at 
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@9-ea/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@9-ea/Method.java:535)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (LUCENE-7407) Explore switching doc values to an iterator API

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499455#comment-15499455
 ] 

David Smiley commented on LUCENE-7407:
--

Awesome progress Mike! It's nice to see we don't have to worry anymore about 
the '0' value being possibly non-existent.

It appears advance(docId) is inherited from DISI and thus it must be called 
with sequential docIDs.  Did you run into any spot in the codebase that didn't 
advance sequentially and so you had to do something different?

> Explore switching doc values to an iterator API
> ---
>
> Key: LUCENE-7407
> URL: https://issues.apache.org/jira/browse/LUCENE-7407
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>  Labels: docValues
> Attachments: LUCENE-7407.patch
>
>
> I think it could be compelling if we restricted doc values to use an
> iterator API at read time, instead of the more general random access
> API we have today:
>   * It would make doc values disk usage more of a "you pay for what
> what you actually use", like postings, which is a compelling
> reduction for sparse usage.
>   * I think codecs could compress better and maybe speed up decoding
> of doc values, even in the non-sparse case, since the read-time
> API is more restrictive "forward only" instead of random access.
>   * We could remove {{getDocsWithField}} entirely, since that's
> implicit in the iteration, and the awkward "return 0 if the
> document didn't have this field" would go away.
>   * We can remove the annoying thread locals we must make today in
> {{CodecReader}}, and close the trappy "I accidentally shared a
> single XXXDocValues instance across threads", since an iterator is
> inherently "use once".
>   * We could maybe leverage the numerous optimizations we've done for
> postings over time, since the two problems ("iterate over doc ids
> and store something interesting for each") are very similar.
> This idea has come up many in the past, e.g. LUCENE-7253 is a recent
> example, and very early iterations of doc values started with exactly
> this ;)
> However, it's a truly enormous change, likely 7.0 only.  Or maybe we
> could have the new iterator APIs also ported to 6.x side by side with
> the deprecate existing random-access APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 851 - Still Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Solaris/851/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

1 tests failed.
FAILED:  org.apache.solr.TestDistributedSearch.test

Error Message:
Expected to find shardAddress in the up shard info: 
{error=org.apache.solr.client.solrj.SolrServerException: No live SolrServers 
available to handle this 
request,trace=org.apache.solr.client.solrj.SolrServerException: No live 
SolrServers available to handle this request  at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:392)
  at 
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:226)
  at 
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:198)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)  at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
 at java.lang.Thread.run(Thread.java:745) ,time=3}

Stack Trace:
java.lang.AssertionError: Expected to find shardAddress in the up shard info: 
{error=org.apache.solr.client.solrj.SolrServerException: No live SolrServers 
available to handle this 
request,trace=org.apache.solr.client.solrj.SolrServerException: No live 
SolrServers available to handle this request
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:392)
at 
org.apache.solr.handler.component.HttpShardHandlerFactory.makeLoadBalancedRequest(HttpShardHandlerFactory.java:226)
at 
org.apache.solr.handler.component.HttpShardHandler.lambda$submit$0(HttpShardHandler.java:198)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
,time=3}
at 
__randomizedtesting.SeedInfo.seed([890E773BB7CBA873:15A48E11937C58B]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at 
org.apache.solr.TestDistributedSearch.comparePartialResponses(TestDistributedSearch.java:1172)
at 
org.apache.solr.TestDistributedSearch.queryPartialResults(TestDistributedSearch.java:1113)
at 
org.apache.solr.TestDistributedSearch.test(TestDistributedSearch.java:973)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsRepeatStatement.callStatement(BaseDistributedSearchTestCase.java:1011)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:960)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 

[jira] [Commented] (SOLR-9526) data_driven configs defaults to "strings" for unmapped fields, makes most fields containing "textual content" unsearchable, breaks tutorial examples

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499254#comment-15499254
 ] 

Shawn Heisey commented on SOLR-9526:


I would think that adding docValues to the field would allow facets to work 
like most people expect, while also allowing single-word searches.

> data_driven configs defaults to "strings" for unmapped fields, makes most 
> fields containing "textual content" unsearchable, breaks tutorial examples
> 
>
> Key: SOLR-9526
> URL: https://issues.apache.org/jira/browse/SOLR-9526
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>
> James Pritchett pointed out on the solr-user list that this sample query from 
> the quick start tutorial matched no docs (even though the tutorial text says 
> "The above request returns only one document")...
> http://localhost:8983/solr/gettingstarted/select?wt=json=true=name:foundation
> The root problem seems to be that the add-unknown-fields-to-the-schema chain 
> in data_driven_schema_configs is configured with...
> {code}
> strings
> {code}
> ...and the "strings" type uses StrField and is not tokenized.
> 
> Original thread: 
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201609.mbox/%3ccac-n2zrpsspfnk43agecspchc5b-0ff25xlfnzogyuvyg2d...@mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [VOTE] Release Lucene/Solr 6.2.1 RC1

[Tomás Fernández Löbbe]

+1

SUCCESS! [0:59:36.664736]

On Fri, Sep 16, 2016 at 6:15 PM, David Smiley 
wrote:

> +1
>
> SUCCESS! [0:54:37.401807]
>
> On Thu, Sep 15, 2016 at 10:08 AM Shalin Shekhar Mangar 
> wrote:
>
>> Please vote for the first release candidate for Lucene/Solr 6.2.1
>>
>> The artifacts can be downloaded from:
>> https://dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.2.1-RC1-
>> rev43ab70147eb494324a1410f7a9f16a896a59bc6f/
>>
>> You can run the smoke tester directly with this command:
>> python3 -u dev-tools/scripts/smokeTestRelease.py https://
>> dist.apache.org/repos/dist/dev/lucene/lucene-solr-6.2.1-RC1-
>> rev43ab70147eb494324a1410f7a9f16a896a59bc6f/
>>
>> Smoke tester passed for me:
>> SUCCESS! [0:29:53.545665]
>>
>> Here's my +1 to release.
>>
>> --
>> Regards,
>> Shalin Shekhar Mangar.
>>
> --
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book: http://www.
> solrenterprisesearchserver.com
>

[jira] [Commented] (SOLR-9524) SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization

[Pushkar Raste (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499190#comment-15499190
 ] 

Pushkar Raste commented on SOLR-9524:
-

I will look through logs of our collections (fortunately we have plenty of 
those) and see how expensive is fingerprint computation. There is already a 
ticket @noble opened for caching fingerprint per segment. I am going to get to 
it next week.

> SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization
> --
>
> Key: SOLR-9524
> URL: https://issues.apache.org/jira/browse/SOLR-9524
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Mike Drob
> Attachments: SOLR-9524.patch
>
>
> In SOLR-9310 we added more code that does some fingerprint caching in 
> SolrIndexSearcher. However, the synchronization looks like it could be made 
> more efficient and may have issues with correctness.
> https://github.com/apache/lucene-solr/blob/branch_6x/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L2371-L2385
> Some of the issues:
> * Double checked locking needs use of volatile variables to ensure proper 
> memory semantics.
> * sync on a ConcurrentHashMap is usually a code smell



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9524) SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization

[Mike Drob (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499123#comment-15499123
 ] 

Mike Drob commented on SOLR-9524:
-

{quote}
RE computeIfAbsent and IOException: I recent reviewed some code and gave 
feedback on an issue pertaining to this (I can't seen to find right it now). 
The code on whatever issue it was could have theoretically used computeIfAbsent 
and would thus have been much nicer were it not for the IOException. So we 
couldn't use it; what a pain, sigh If we want to use it here too (were it 
not for IOException) we might want to add a utility to make this easy so that 
the semantics of what we're doing is clear. computeIfAbsent is beautiful when 
you can use it.
{quote}
I've been poking at this, but due to some quirks of java generics, it's harder 
to put in a generic utility than it would appear at first blush. Going down 
this route, we may have to limit ourselves strictly to IOException.

bq. I don't think it's a big deal. Essentially it's just made caching useless. 
We didn't have any caching till now. So, it is not going to be any worse than 
what it used to be
It only made the second check useless. The first check and the put are still 
correct, so we will still have some amount of caching right now, but multiple 
threads asking at once could end up all doing the work.

> SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization
> --
>
> Key: SOLR-9524
> URL: https://issues.apache.org/jira/browse/SOLR-9524
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Mike Drob
> Attachments: SOLR-9524.patch
>
>
> In SOLR-9310 we added more code that does some fingerprint caching in 
> SolrIndexSearcher. However, the synchronization looks like it could be made 
> more efficient and may have issues with correctness.
> https://github.com/apache/lucene-solr/blob/branch_6x/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L2371-L2385
> Some of the issues:
> * Double checked locking needs use of volatile variables to ensure proper 
> memory semantics.
> * sync on a ConcurrentHashMap is usually a code smell



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9524) SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15499113#comment-15499113
 ] 

Yonik Seeley commented on SOLR-9524:


bq. We didn't have any caching till now. So, it is not going to be any worse 
than what it used to be

It was cached before SOLR-9310, so if everyone was in sync (say coming back up 
from a reboot, or after electing a new leader), it could certainly be more 
expensive than it was.


> SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization
> --
>
> Key: SOLR-9524
> URL: https://issues.apache.org/jira/browse/SOLR-9524
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Mike Drob
> Attachments: SOLR-9524.patch
>
>
> In SOLR-9310 we added more code that does some fingerprint caching in 
> SolrIndexSearcher. However, the synchronization looks like it could be made 
> more efficient and may have issues with correctness.
> https://github.com/apache/lucene-solr/blob/branch_6x/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L2371-L2385
> Some of the issues:
> * Double checked locking needs use of volatile variables to ensure proper 
> memory semantics.
> * sync on a ConcurrentHashMap is usually a code smell



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-6.2-Linux (64bit/jdk1.8.0_102) - Build # 26 - Still unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-6.2-Linux/26/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test

Error Message:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:40976/_/c8n_1x3_lf_shard1_replica3]

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: 
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request:[http://127.0.0.1:40976/_/c8n_1x3_lf_shard1_replica3]
at 
__randomizedtesting.SeedInfo.seed([BE1E837B887A1E97:364ABCA12686736F]:0)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.directUpdate(CloudSolrClient.java:769)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1161)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1050)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:992)
at org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:592)
at 
org.apache.solr.cloud.HttpPartitionTest.sendDoc(HttpPartitionTest.java:578)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.testRf3WithLeaderFailover(LeaderFailoverAfterPartitionTest.java:174)
at 
org.apache.solr.cloud.LeaderFailoverAfterPartitionTest.test(LeaderFailoverAfterPartitionTest.java:55)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:871)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:907)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:921)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:992)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:967)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:809)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:460)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:880)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:781)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:816)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:827)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

[jira] [Updated] (LUCENE-7407) Explore switching doc values to an iterator API

[Michael McCandless (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-7407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael McCandless updated LUCENE-7407:
---
Attachment: LUCENE-7407.patch

I think this nearly ready!  I've fixed all nocommits, but {{ant precommit}} is 
a bit angry still... I'll fix before pushing.

I'm attaching an applyable patch vs. current master.

All doc values usage has been switched to iterators instead of random access, 
and {{getDocsWithField}} is gone.

I've done very little to improve the default codec to take advantage of this.  
I think there is a lot of fun improvements we can make here, in follow-on 
issues, so that e.g. LUCENE-7253 (merging of sparse doc values fields) is fixed.

To write doc values we now pass a {{DocValuesProducer}} (instead of N 
Iterables), and I created legacy deprecated bridge classes 
({{LegacyDocValuesIterables}}) to turns these back into Iterables for existing 
codecs.

I also created legacy bridge classes to turn random access DVs into the new 
iterators.


> Explore switching doc values to an iterator API
> ---
>
> Key: LUCENE-7407
> URL: https://issues.apache.org/jira/browse/LUCENE-7407
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>  Labels: docValues
> Attachments: LUCENE-7407.patch
>
>
> I think it could be compelling if we restricted doc values to use an
> iterator API at read time, instead of the more general random access
> API we have today:
>   * It would make doc values disk usage more of a "you pay for what
> what you actually use", like postings, which is a compelling
> reduction for sparse usage.
>   * I think codecs could compress better and maybe speed up decoding
> of doc values, even in the non-sparse case, since the read-time
> API is more restrictive "forward only" instead of random access.
>   * We could remove {{getDocsWithField}} entirely, since that's
> implicit in the iteration, and the awkward "return 0 if the
> document didn't have this field" would go away.
>   * We can remove the annoying thread locals we must make today in
> {{CodecReader}}, and close the trappy "I accidentally shared a
> single XXXDocValues instance across threads", since an iterator is
> inherently "use once".
>   * We could maybe leverage the numerous optimizations we've done for
> postings over time, since the two problems ("iterate over doc ids
> and store something interesting for each") are very similar.
> This idea has come up many in the past, e.g. LUCENE-7253 is a recent
> example, and very early iterations of doc values started with exactly
> this ;)
> However, it's a truly enormous change, likely 7.0 only.  Or maybe we
> could have the new iterator APIs also ported to 6.x side by side with
> the deprecate existing random-access APIs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498680#comment-15498680
 ] 

Noble Paul commented on SOLR-9512:
--

So in your solution here is what happens

# Instead of just passing one server , we pass all the nodes to 
LBHttpSolrClient (LBHSC). The shard leader should be the first in the list
# LBHSC knows that the leader is a dead node (or it will soon know that). So it 
would pick up the next server in the list and makes a request there
# The request would come back with an error (no leader)
# CSC returns the call with an error "no leader"

Is that right?

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498631#comment-15498631
 ] 

Alan Woodward commented on SOLR-9512:
-

(apologies for the terrible metaphor there, it's early on a Saturday)

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498629#comment-15498629
 ] 

Alan Woodward commented on SOLR-9512:
-

It's kicking the can to a live server, though, which will be able to tell us 
either a) yes, everything's fine now, you just need to invalidate your cache, 
or b) there's no leader for this shard, so fail the update.  Either way, we get 
useful information, whereas at the moment we're trying to find out what's 
happening from a dead server.

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_102) - Build # 6122 - Still Unstable!

[Policeman Jenkins Server]

Build: http://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/6122/
Java: 64bit/jdk1.8.0_102 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.CdcrVersionReplicationTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! [InternalHttpClient]

Stack Trace:
java.lang.AssertionError: ObjectTracker found 1 object(s) that were not 
released!!! [InternalHttpClient]
at __randomizedtesting.SeedInfo.seed([91FB3E0E72DC09B8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.SolrTestCaseJ4.teardownTestCases(SolrTestCaseJ4.java:257)
at sun.reflect.GeneratedMethodAccessor20.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1764)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:834)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:367)
at java.lang.Thread.run(Thread.java:745)


FAILED:  org.apache.solr.handler.TestReplicationHandlerBackup.doTestBackup

Error Message:
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup_91FB3E0E72DC09B8-001\solr-instance-001\collection1\data\snapshot.cbzjndqbdfqokxbro\_0.fnm

Stack Trace:
java.nio.file.NoSuchFileException: 
C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.handler.TestReplicationHandlerBackup_91FB3E0E72DC09B8-001\solr-instance-001\collection1\data\snapshot.cbzjndqbdfqokxbro\_0.fnm
at 
__randomizedtesting.SeedInfo.seed([91FB3E0E72DC09B8:D0701E6B5562FAF7]:0)
at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:79)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:102)
at 
sun.nio.fs.WindowsFileSystemProvider.newByteChannel(WindowsFileSystemProvider.java:230)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at 
org.apache.lucene.store.SimpleFSDirectory.openInput(SimpleFSDirectory.java:77)
at 
org.apache.lucene.store.Directory.openChecksumInput(Directory.java:119)
at 
org.apache.lucene.codecs.lucene60.Lucene60FieldInfosFormat.read(Lucene60FieldInfosFormat.java:113)
at 
org.apache.lucene.index.SegmentCoreReaders.(SegmentCoreReaders.java:101)
at org.apache.lucene.index.SegmentReader.(SegmentReader.java:74)
at 
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62)
at 
org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:54)
at 
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:659)
at 

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498506#comment-15498506
 ] 

Noble Paul commented on SOLR-9512:
--

bq. there is a leader, it's just that we locally have the wrong one cached

When we get an error , we must invalidate the cache. That's part of the 
solution. But the larger problem is that leader election takes a while and the 
state.json will have that information after sometime. The solution of sending 
the doc to another replica in the shard is just kicking the can down the road

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498495#comment-15498495
 ] 

Alan Woodward commented on SOLR-9512:
-

I don't think so - there *is* a leader, it's just that we locally have the 
wrong one cached.  And because we're sending our updates *only* to the previous 
(and now down) leader, we get a failure, when instead we ought to be sending 
this update on to the next leader.

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated SOLR-9512:

Comment: was deleted

(was: I don't think so - there *is* a leader, it's just that we locally have 
the wrong one cached.  And because we're sending our updates *only* to the 
previous (and now down) leader, we get a failure, when instead we ought to be 
sending this update on to the next leader.)

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498496#comment-15498496
 ] 

Alan Woodward commented on SOLR-9512:
-

I don't think so - there *is* a leader, it's just that we locally have the 
wrong one cached.  And because we're sending our updates *only* to the previous 
(and now down) leader, we get a failure, when instead we ought to be sending 
this update on to the next leader.

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9512) CloudSolrClient's cluster state cache can break direct updates to leaders

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498485#comment-15498485
 ] 

Noble Paul commented on SOLR-9512:
--

It's just hiding the problem. The fact that there is no leader for that shard 
means that the request will fail eventually. 

> CloudSolrClient's cluster state cache can break direct updates to leaders
> -
>
> Key: SOLR-9512
> URL: https://issues.apache.org/jira/browse/SOLR-9512
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Alan Woodward
>
> This is the root cause of SOLR-9305 and (at least some of) SOLR-9390.  The 
> process goes something like this:
> Documents are added to the cluster via a CloudSolrClient, with 
> directUpdatesToLeadersOnly set to true.  CSC caches its view of the 
> DocCollection.  The leader then goes down, and is reassigned.  Next time 
> documents are added, CSC checks its cache again, and gets the old view of the 
> DocCollection.  It then tries to send the update directly to the old, now 
> down, leader, and we get ConnectionRefused.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-9523) Refactor CoreAdminOperation into smaller classes

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-9523.
--
   Resolution: Fixed
Fix Version/s: 6.3

> Refactor CoreAdminOperation into smaller classes
> 
>
> Key: SOLR-9523
> URL: https://issues.apache.org/jira/browse/SOLR-9523
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
> Fix For: 6.3
>
>
> It's too big and unreadable



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9493) uniqueKey generation fails if content POSTed as "application/javabin" and uniqueKey field comes as NULL (as opposed to not coming at all).

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498475#comment-15498475
 ] 

Noble Paul commented on SOLR-9493:
--

bq.but just because as you said "ID is very important in SolrCloud", I wanted 
SOLR itself to generate that ID

You missed the point.
SolrJ is designed to send a doc to the right leader by hashing the id. If Solr 
is assigns the id, SolrJ has no clue where to send this to. So, the doc is sent 
to a random node. The node assigns the id and it realizes that the doc belongs 
to another shard and it is forwarded there. Essentially, you lost the 
efficiency of using SolrJ by asking Solr to assign an ID.

> uniqueKey generation fails if content POSTed as "application/javabin" and 
> uniqueKey field comes as NULL (as opposed to not coming at all).
> --
>
> Key: SOLR-9493
> URL: https://issues.apache.org/jira/browse/SOLR-9493
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Yury Kartsev
> Attachments: 200.png, 400.png, Screen Shot 2016-09-11 at 16.29.50 
> .png, SolrInputDoc_contents.png, SolrInputDoc_headers.png
>
>
> I have faced a weird issue when the same application code (using SolrJ) fails 
> indexing a document without a unique key (should be auto-generated by SOLR) 
> in SolrCloud and succeeds indexing it in standalone SOLR instance (or even in 
> cloud mode, but from web interface of one of the replicas). Difference is 
> obviously only between clients (CloudSolrClient vs HttpSolrClient) and SOLR 
> URLs (Zokeeper hostname+port vs standalone SOLR instance hostname and port). 
> Failure is seen as "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id".
> I am using SOLR 5.1. In cloud mode I have 1 shard and 3 replicas.
> After lot of debugging and investigation (see below as well as my 
> [StackOverflow 
> post|http://stackoverflow.com/questions/39401792/uniquekey-generation-does-not-work-in-solrcloud-but-works-if-standalone])
>  I came to a conclusion that the difference in failing and succeeding calls 
> is simply content type of the POSTing requests. Local proxy clearly shows 
> that the request fails if content is sent as "application/javabin" (see 
> attached screenshot with sensitive data removed) and succeeds if content sent 
> as "application/xml; charset=UTF-8"  (see attached screenshot with sensitive 
> data removed).
> Would you be able to please assist?
> Thank you very much in advance!
> 
> Copying whole description and investigation here as well:
> 
> [Documentation|https://cwiki.apache.org/confluence/display/solr/Other+Schema+Elements]
>  states:{quote}Schema defaults and copyFields cannot be used to populate the 
> uniqueKey field. You can use UUIDUpdateProcessorFactory to have uniqueKey 
> values generated automatically.{quote}
> Therefore I have added my uniqueKey field to the schema:{code} name="uuid" class="solr.UUIDField" indexed="true" />
> ...
> 
> ...
> id{code}Then I have added updateRequestProcessorChain 
> to my solrconfig:{code}
> 
> id
> 
> 
> {code}And made it the default for the 
> UpdateRequestHandler:{code}
>  
>   uuid
>  
> {code}
> Adding new documents with null/absent id works fine as from web-interface of 
> one of the replicas, as when using SOLR in standalone mode (non-cloud) from 
> my application. Although when only I'm using SolrCloud and add document from 
> my application (using CloudSolrClient from SolrJ) it fails with 
> "org.apache.solr.client.solrj.SolrServerException: 
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: 
> Document is missing mandatory uniqueKey field: id"
> All other operations like ping or search for documents work fine in either 
> mode (standalone or cloud).
> INVESTIGATION (i.e. more details):
> In standalone mode obviously update request is:{code}POST 
> standalone_host:port/solr/collection_name/update?wt=json{code}
> In SOLR cloud mode, when adding document from one replica's web interface, 
> update request is (found through inspecting the call made by web interface): 
> {code}POST 
> replica_host:port/solr/collection_name_shard1_replica_1/update?wt=json{code}
> In both these cases payload is something like:{code}{
> "add": {
> "doc": {
>  .
> },
> "boost": 1.0,
> "overwrite": true,
> "commitWithin": 1000
> }
> }{code}
> In case when CloudSolrClient is used, the following happens (found through 
> debugging):
> Using ZK and some logic, URL list of replicas is 

[jira] [Commented] (SOLR-9523) Refactor CoreAdminOperation into smaller classes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498460#comment-15498460
 ] 

ASF subversion and git services commented on SOLR-9523:
---

Commit 286b35b0200a84cd734073574081195987b6e236 in lucene-solr's branch 
refs/heads/master from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=286b35b ]

SOLR-9523: Refactor CoreAdminOperation into smaller classes


> Refactor CoreAdminOperation into smaller classes
> 
>
> Key: SOLR-9523
> URL: https://issues.apache.org/jira/browse/SOLR-9523
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> It's too big and unreadable



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9523) Refactor CoreAdminOperation into smaller classes

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498456#comment-15498456
 ] 

ASF subversion and git services commented on SOLR-9523:
---

Commit 68d9d97510c8c46992cca06c0874cbe0169cdd22 in lucene-solr's branch 
refs/heads/branch_6x from [~noble.paul]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=68d9d97 ]

SOLR-9523: Refactor CoreAdminOperation into smaller classes


> Refactor CoreAdminOperation into smaller classes
> 
>
> Key: SOLR-9523
> URL: https://issues.apache.org/jira/browse/SOLR-9523
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Assignee: Noble Paul
>
> It's too big and unreadable



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-9524) SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-9524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15498410#comment-15498410
 ] 

Noble Paul commented on SOLR-9524:
--

 bq. fingerprint = maxVersionFingerprintCache.get(maxVersionFingerprintCache);

LOL, nice bug.


bq. May be we should hold off 6.2.1 release

[~praste] I don't think it's a big deal. Essentially it's just made caching 
useless. We didn't have any caching till now. So, it is not going to be any 
worse than what it used to be 

> SolrIndexSearcher.getIndexFingerprint uses dubious sunchronization
> --
>
> Key: SOLR-9524
> URL: https://issues.apache.org/jira/browse/SOLR-9524
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.3
>Reporter: Mike Drob
> Attachments: SOLR-9524.patch
>
>
> In SOLR-9310 we added more code that does some fingerprint caching in 
> SolrIndexSearcher. However, the synchronization looks like it could be made 
> more efficient and may have issues with correctness.
> https://github.com/apache/lucene-solr/blob/branch_6x/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java#L2371-L2385
> Some of the issues:
> * Double checked locking needs use of volatile variables to ensure proper 
> memory semantics.
> * sync on a ConcurrentHashMap is usually a code smell



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org