[jira] [Commented] (SOLR-12028) BadApple and AwaitsFix annotations usage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464630#comment-16464630
 ] 

ASF subversion and git services commented on SOLR-12028:


Commit 2558a06f3c8040d47de6606da2539532e4c67854 in lucene-solr's branch 
refs/heads/branch_7x from Erick Erickson
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2558a06 ]

SOLR-12028: BadApple and AwaitsFix annotations usage

(cherry picked from commit 89fc02a)


> BadApple and AwaitsFix annotations usage
> 
>
> Key: SOLR-12028
> URL: https://issues.apache.org/jira/browse/SOLR-12028
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12016-buildsystem.patch, SOLR-12028-3-Mar.patch, 
> SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch
>
>
> There's a long discussion of this topic at SOLR-12016. Here's a summary:
> - BadApple annotations are used for tests that intermittently fail, say < 30% 
> of the time. Tests that fail more often shold be moved to AwaitsFix. This is, 
> of course, a judgement call
> - AwaitsFix annotations are used for tests that, for some reason, the problem 
> can't be fixed immediately. Likely reasons are third-party dependencies, 
> extreme difficulty tracking down, dependency on another JIRA etc.
> Jenkins jobs will typically run with BadApple disabled to cut down on noise. 
> Periodically Jenkins jobs will be run with BadApples enabled so BadApple 
> tests won't be lost and reports can be generated. Tests that run with 
> BadApples disabled that fail require _immediate_ attention.
> The default for developers is that BadApple is enabled.
> If you are working on one of these tests and cannot get the test to fail 
> locally, it is perfectly acceptable to comment the annotation out. You should 
> let the dev list know that this is deliberate.
> This JIRA is a placeholder for BadApple tests to point to between the times 
> they're identified as BadApple and they're either fixed or changed to 
> AwaitsFix or assigned their own JIRA.
> I've assigned this to myself to track so I don't lose track of it. No one 
> person will fix all of these issues, this will be an ongoing technical debt 
> cleanup effort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12028) BadApple and AwaitsFix annotations usage

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464629#comment-16464629
 ] 

ASF subversion and git services commented on SOLR-12028:


Commit 89fc02a3b04aedf2803a46782b3a8b8e9b5c9a7b in lucene-solr's branch 
refs/heads/master from Erick Erickson
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=89fc02a ]

SOLR-12028: BadApple and AwaitsFix annotations usage


> BadApple and AwaitsFix annotations usage
> 
>
> Key: SOLR-12028
> URL: https://issues.apache.org/jira/browse/SOLR-12028
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Tests
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: SOLR-12016-buildsystem.patch, SOLR-12028-3-Mar.patch, 
> SOLR-12028-sysprops-reproduce.patch, SOLR-12028.patch, SOLR-12028.patch
>
>
> There's a long discussion of this topic at SOLR-12016. Here's a summary:
> - BadApple annotations are used for tests that intermittently fail, say < 30% 
> of the time. Tests that fail more often shold be moved to AwaitsFix. This is, 
> of course, a judgement call
> - AwaitsFix annotations are used for tests that, for some reason, the problem 
> can't be fixed immediately. Likely reasons are third-party dependencies, 
> extreme difficulty tracking down, dependency on another JIRA etc.
> Jenkins jobs will typically run with BadApple disabled to cut down on noise. 
> Periodically Jenkins jobs will be run with BadApples enabled so BadApple 
> tests won't be lost and reports can be generated. Tests that run with 
> BadApples disabled that fail require _immediate_ attention.
> The default for developers is that BadApple is enabled.
> If you are working on one of these tests and cannot get the test to fail 
> locally, it is perfectly acceptable to comment the annotation out. You should 
> let the dev list know that this is deliberate.
> This JIRA is a placeholder for BadApple tests to point to between the times 
> they're identified as BadApple and they're either fixed or changed to 
> AwaitsFix or assigned their own JIRA.
> I've assigned this to myself to track so I don't lose track of it. No one 
> person will fix all of these issues, this will be an ongoing technical debt 
> cleanup effort.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk-9.0.4) - Build # 1850 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1850/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseG1GC

10 tests failed.
FAILED:  
org.apache.solr.cloud.LeaderVoteWaitTimeoutTest.testMostInSyncReplicasCanWinElection

Error Message:
Timeout waiting for new leader null Live Nodes: [127.0.0.1:38109_solr, 
127.0.0.1:41517_solr, 127.0.0.1:43795_solr] Last available state: 
DocCollection(collection1//collections/collection1/state.json/14)={   
"pullReplicas":"0",   "replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node62":{   "core":"collection1_shard1_replica_n61",   
"base_url":"https://127.0.0.1:43229/solr;,   
"node_name":"127.0.0.1:43229_solr",   "state":"down",   
"type":"NRT",   "force_set_state":"false"}, "core_node64":{ 
  "core":"collection1_shard1_replica_n63",   
"base_url":"https://127.0.0.1:43795/solr;,   
"node_name":"127.0.0.1:43795_solr",   "state":"active",   
"type":"NRT",   "force_set_state":"false"}, "core_node66":{ 
  "core":"collection1_shard1_replica_n65",   
"base_url":"https://127.0.0.1:41517/solr;,   
"node_name":"127.0.0.1:41517_solr",   "state":"active",   
"type":"NRT",   "force_set_state":"false",   
"router":{"name":"compositeId"},   "maxShardsPerNode":"1",   
"autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Timeout waiting for new leader
null
Live Nodes: [127.0.0.1:38109_solr, 127.0.0.1:41517_solr, 127.0.0.1:43795_solr]
Last available state: 
DocCollection(collection1//collections/collection1/state.json/14)={
  "pullReplicas":"0",
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node62":{
  "core":"collection1_shard1_replica_n61",
  "base_url":"https://127.0.0.1:43229/solr;,
  "node_name":"127.0.0.1:43229_solr",
  "state":"down",
  "type":"NRT",
  "force_set_state":"false"},
"core_node64":{
  "core":"collection1_shard1_replica_n63",
  "base_url":"https://127.0.0.1:43795/solr;,
  "node_name":"127.0.0.1:43795_solr",
  "state":"active",
  "type":"NRT",
  "force_set_state":"false"},
"core_node66":{
  "core":"collection1_shard1_replica_n65",
  "base_url":"https://127.0.0.1:41517/solr;,
  "node_name":"127.0.0.1:41517_solr",
  "state":"active",
  "type":"NRT",
  "force_set_state":"false",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
at 
__randomizedtesting.SeedInfo.seed([B311C8644DEBF4A7:1B0DD4DE8FABC08D]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.SolrCloudTestCase.waitForState(SolrCloudTestCase.java:278)
at 
org.apache.solr.cloud.LeaderVoteWaitTimeoutTest.testMostInSyncReplicasCanWinElection(LeaderVoteWaitTimeoutTest.java:187)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 

[jira] [Commented] (SOLR-12314) ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the solr.xml file

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464583#comment-16464583
 ] 

Varun Thacker commented on SOLR-12314:
--

Yeah as soon as you mentioned this on SOLR-11881 I knew this patch wasn't the 
right approach
{quote}Cool - the tricky part is, if only one of the properties of the two is 
overridden on the client itself on the fly, we still want to pick up the 
default for the other one.
{quote}

> ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the 
> solr.xml file
> -
>
> Key: SOLR-12314
> URL: https://issues.apache.org/jira/browse/SOLR-12314
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12314.patch
>
>
> In ConcurrentUpdateSolrClient we create an HttpPost Request which allows you 
> to set a request config. If the request config is not provided httpclient 
> will use the default request config. 
>  
> {code:java}
> org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
> HttpClientUtil.createDefaultRequestConfigBuilder();
> if (soTimeout != null) {
>   requestConfigBuilder.setSocketTimeout(soTimeout);
> }
> if (connectionTimeout != null) {
>   requestConfigBuilder.setConnectTimeout(connectionTimeout);
> }
> method.setConfig(requestConfigBuilder.build());{code}
> While creating the httpclient object we ensure that the default request is 
> set with the properties we care about.  This happens in 
> HttpClientUtils#setupBuilder
> {code:java}
> RequestConfig requestConfig = requestConfigBuilder.build();
> HttpClientBuilder retBuilder = 
> builder.setDefaultRequestConfig(requestConfig);{code}
> So there is no need to set a per request config 
>  
> Here is where the httpclient picks the request config is provided on the 
> request itself : 
> [https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L168]
>  
> And if it's not provided it uses the default here : 
> https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L148



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11934) Visit Solr logging, it's too noisy.

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464575#comment-16464575
 ] 

Erick Erickson commented on SOLR-11934:
---

I looked at sample logs from a client and wrote a little program to bucket 
them. NOTE: this was an index-heavy type application. Here are the number of 
log messages originating in these files:

35 occurrences in 
org.apache.solr.common.cloud.ZkStateReader$StateWatcher;
48 occurrences in org.apache.solr.core.QuerySenderListener;
55 occurrences in org.apache.solr.servlet.HttpSolrCall;
 64388 occurrences in org.apache.solr.update.SolrIndexWriter;
 64413 occurrences in org.apache.solr.search.SolrIndexSearcher;
 64601 occurrences in org.apache.solr.core.SolrCore;
128795 occurrences in org.apache.solr.update.DirectUpdateHandler2;
   3345415 occurrences in 
org.apache.solr.update.processor.LogUpdateProcessorFactory$LogUpdateProcessor;

that got me to thinking about the _value_ of certain messages when 
troubleshooting. Is the value in the individual messages or in getting a sense 
of how things are behaving?

So WDYT about logging some summary information instead of individual messages 
for high-cardinality messages? Rather than logging every update at info level, 
would it be "useful enough" when troubleshooting to log a summary of updates' 
vital statistics every N seconds?

Here's what I'm thinking. Pretty often when troubleshooting, we want to 
correlate heavy request activity for Solr to something else (GC, slowdowns, 
recoveries, whatever). Using updates as an example, what if at info level, we 
logged something like

INFO  - 2018-05-04 03:35:09.206 over the last T seconds there were N update 
requests updating D documents taking min/avg/median/max milliseconds to 
complete with min/average/median/max documents per request.

With the new metrics capabilities, that information may be available already, 
along with lots of other information (percentiles and the like).

If seeing the underlying individual requests were important, one could turn on 
finer-grained logging for this class only. We could even supply commented-out 
ways to enable voluminous messages for anything we choose to write summary 
information out in the log4j2 conifgs.

I totally understand that for gnarly problems you want (and need) voluminous 
output. I totally understand that sometimes you can't predict that up front. I 
totally understand that in some cases this would introduce an extra round-trip, 
"you have to enable this option in your logging configuration, wait for it to 
happen again and call me".

I also totally understand that in large Solr operations, the log files are 
very, very hard to use because there's so much information being dumped by 
potentially hundreds of Solr instances.

I also totally understand that I've spend more time than I like to remember 
writing programs to parse log files to give me summary information like the 
above to know where to start looking for problems. Why not have Solr do it?

This example is just for updates, but queries could do something very similar 
along with whatever else fits this pattern. Queries certainly come to mind.

The question I'm really asking is if we're logging intelligently or not. 
Basically we're throwing everything that anyone _might_ be interested into a 
log file and then requiring the users (and consultants) to figure out how to 
make it useful. We know enough about how this information is used I think to do 
some of it up front, always with the option to drop back to the finer-grained 
information when necessary.

Anyway, what do people think?

> Visit Solr logging, it's too noisy.
> ---
>
> Key: SOLR-11934
> URL: https://issues.apache.org/jira/browse/SOLR-11934
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> I think we have way too much INFO level logging. Or, perhaps more correctly, 
> Solr logging needs to be examined and messages logged at an appropriate level.
> We log every update at an INFO level for instance. But I think we log LIR at 
> INFO as well. As a sysadmin I don't care to have my logs polluted with a 
> message for every update, but if I'm trying to keep my system healthy I want 
> to see LIR messages and try to understand why.
> Plus, in large installations logging at INFO level is creating a _LOT_ of 
> files.
> What I want to discuss on this JIRA is
> 1> What kinds of messages do we want log at WARN, INFO, DEBUG, and TRACE 
> levels?
> 2> Who's the audience at each level? For a running system that's functioning, 
> sysops folks would really like WARN messages that mean something need 
> attention for instance. If 

[jira] [Commented] (SOLR-12314) ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the solr.xml file

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464573#comment-16464573
 ] 

Mark Miller commented on SOLR-12314:


We can't just remove it though. The CUSC timeout setters still need to work 
until they are removed.

They also need to work if set one or both of them - if neither are set, we can 
just skip the request config object, if both are set, we use it, and if one is 
set we have to get the right default for the other to set on the request config.

I think this also breaks the time outs coming from the builder if set that way.

If you set timeouts on the client itself, they should work - it should only 
come from elsewhere or a passed in HttpClient object if they are not explicitly 
set on the client itself.

> ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the 
> solr.xml file
> -
>
> Key: SOLR-12314
> URL: https://issues.apache.org/jira/browse/SOLR-12314
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12314.patch
>
>
> In ConcurrentUpdateSolrClient we create an HttpPost Request which allows you 
> to set a request config. If the request config is not provided httpclient 
> will use the default request config. 
>  
> {code:java}
> org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
> HttpClientUtil.createDefaultRequestConfigBuilder();
> if (soTimeout != null) {
>   requestConfigBuilder.setSocketTimeout(soTimeout);
> }
> if (connectionTimeout != null) {
>   requestConfigBuilder.setConnectTimeout(connectionTimeout);
> }
> method.setConfig(requestConfigBuilder.build());{code}
> While creating the httpclient object we ensure that the default request is 
> set with the properties we care about.  This happens in 
> HttpClientUtils#setupBuilder
> {code:java}
> RequestConfig requestConfig = requestConfigBuilder.build();
> HttpClientBuilder retBuilder = 
> builder.setDefaultRequestConfig(requestConfig);{code}
> So there is no need to set a per request config 
>  
> Here is where the httpclient picks the request config is provided on the 
> request itself : 
> [https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L168]
>  
> And if it's not provided it uses the default here : 
> https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L148



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8207) Modernise cloud tab on Admin UI

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8207?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464572#comment-16464572
 ] 

David Smiley commented on SOLR-8207:


Looking very nice Jan!  It's sweet to see this info being made so accessible.
 * the large fonts for CPU/Heap/Disk seem uncalled for; it gives the appearance 
that it's super important and maybe trying to tell me about a problem
 * it's not visually evident that clicking stuff will do something.

> Modernise cloud tab on Admin UI
> ---
>
> Key: SOLR-8207
> URL: https://issues.apache.org/jira/browse/SOLR-8207
> Project: Solr
>  Issue Type: Improvement
>  Components: Admin UI
>Affects Versions: 5.3
>Reporter: Upayavira
>Assignee: Jan Høydahl
>Priority: Major
> Attachments: node-compact.png, node-details.png, node-hostcolumn.png, 
> node-toggle-row-numdocs.png, nodes-tab-real.png, nodes-tab.png
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The various sub-tabs of the "Cloud tab" were designed before anyone was 
> making real use of SolrCloud, and when we didn't really know the use-cases we 
> would need to support. I would argue that, whilst they are pretty (and 
> clever) they aren't really fit for purpose (with the exception of tree view).
> Issues:
> * Radial view doesn't scale beyond a small number of nodes/collections
> * Paging on the graph view is based on collections - so a collection with 
> many replicas won't be subject to pagination
> * The Dump feature is kinda redundant and should be removed
> * There is now a major overlap in functionality with the new Collections tab
> What I'd propose is that we:
>  * promote the tree tab to top level
>  * remove the graph views and the dump tab
>  * add a new Nodes tab
> This nodes tab would complement the collections tab - showing nodes, and 
> their associated replicas/collections. From this view, it would be possible 
> to add/remove replicas and to see the status of nodes. It would also be 
> possible to filter nodes by status: "show me only up nodes", "show me nodes 
> that are in trouble", "show me nodes that have leaders on them", etc.
> Presumably, if we have APIs to support it, we might have a "decommission 
> node" option, that would ensure that no replicas on this node are leaders, 
> and then remove all replicas from the node, ready for it to be removed from 
> the cluster.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12292) Make it easier to configure Solr with CORS

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464569#comment-16464569
 ] 

David Smiley commented on SOLR-12292:
-

Even with POST, what good would that do to a SearchHandler?

> Make it easier to configure Solr with CORS
> --
>
> Key: SOLR-12292
> URL: https://issues.apache.org/jira/browse/SOLR-12292
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Priority: Major
>
> While working on SOLR-8207 I wanted to collect info from other SolrCloud 
> nodes from the AdminUI. However this is blocked by 
> [CORS|https://en.wikipedia.org/wiki/Cross-origin_resource_sharing] policy. In 
> that Jira I instead did the fan-out on the Solr server side for the two 
> handler I needed.
> It would be nice if all nodes in a SolrCloud cluster could automatically 
> accept any other node as a legal origin, and make it easy for users to add 
> other origins by config.
> If we use the [Jetty CORS 
> filter|http://www.eclipse.org/jetty/documentation/9.4.9.v20180320/cross-origin-filter.html]
>  in web.xml, perhaps we could parse a env.var from solr.in.xx and inject into 
> the {{allowedOrigins}} property of that filter? There is also SOLR-6059 which 
> tries to implement CORS inside of Solr handlers and not in Jetty. Don't know 
> pros/cons of those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12297) Create a good SolrClient for SolrCloud paving the way for async requests, HTTP2, multiplexing, and the latest & greatest Jetty features.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464567#comment-16464567
 ] 

Mark Miller commented on SOLR-12297:


bq. Our clients don't support talking to HTTP2 so I'm not sure how that would 
actually work unless it was single server Solr with no internal communication. 

Ah, you actually could get this to work with our clients, they just wouldn't 
use HTTP2. I didn't know that you can serve HTTP1.1 and HTTP2 on the same port. 
You have to use ALPN though and that is kind of a mess before Java 9. Good to 
know though, that is a much nicer option for our transition to support HTTP2 
than running two connectors on two ports.

> Create a good SolrClient for SolrCloud paving the way for async requests, 
> HTTP2, multiplexing, and the latest & greatest Jetty features.
> 
>
> Key: SOLR-12297
> URL: https://issues.apache.org/jira/browse/SOLR-12297
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Priority: Major
>
> Blocking or async support as well as HTTP2 compatible with multiplexing.
> Once it supports enough and is stable, replace internal usage, allowing 
> async, and eventually move to HTTP2 connector and allow multiplexing. Could 
> support HTTP1.1 and HTTP2 on different ports depending on state of the world 
> then.
> The goal of the client itself is to work against HTTP1.1 or HTTP2 with 
> minimal or no code path differences and the same for async requests (should 
> initially work for both 1.1 and 2 and share majority of code).
> The client should also be able to replace HttpSolrClient and plug into the 
> other clients the same way.
> I doubt it would make sense to keep ConcurrentUpdateSolrClient eventually 
> though.
> I evaluated some clients and while there are a few options, I went with 
> Jetty's HttpClient. It's more mature than Apache HttpClient's support (in 5 
> beta) and we would have to update to a new API for Apache HttpClient anyway.
> Meanwhile, the Jetty guys have been very supportive of helping Solr with any 
> issues and I like having the client and server from the same project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11884) find/fix inefficiencies in our use of logging

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464564#comment-16464564
 ] 

Erick Erickson commented on SOLR-11884:
---

Not yet. We're going through the uses of log.info("message" + blahblahblah) 
type calls. Although that could be another JIRA.

> find/fix inefficiencies in our use of logging
> -
>
> Key: SOLR-11884
> URL: https://issues.apache.org/jira/browse/SOLR-11884
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: logging
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> We've been looking at Solr using Flight Recorder and ran across some 
> interesting things I'd like to discuss. Let's discuss general logging 
> approaches here, then perhaps break out sub-JIRAs when we reach any kind of 
> agreement.
> 1> Every log message generates a new Throwable, presumably to get things like 
> line number, file, class name and the like. On a 2 minute run blasting 
> updates this meant 150,000 (yes, 150K) instances of "new Throwable()".
>  
> See the section "Asynchronous Logging with Caller Location Information" at:
> [https://logging.apache.org/log4j/2.x/performance.html]
> I'm not totally sure changing the layout pattern will fix this in log4j 1.x, 
> but apparently certainly should in log4j 2.
>  
> The cost of course would be that lots of our log messages would lack some of 
> the information. Exceptions would still contain all the file/class/line 
> information of course.
>  
> Proposal:
> Change the layout pattern to, by default, _NOT_  include information that 
> requires a Throwable to be created. Also include a pattern that could be 
> un-commented to get this information back for troubleshooting.
>  
> 
>  
> We generate strings when we don't need them. Any construct like
> log.info("whatever " + method_that_builds_a_string + " : " + some_variable);
> generates the string (some of which are quite expensive) and then throws it 
> away if the log level is at, say, WARN. The above link also shows that 
> parameterizing this doesn't suffer this problem, so anything like the above 
> should be re-written as:
> log.info("whatever {} : {} ", method_that_builds_a_string, some_variable);
>  
> The alternative is to do something like but let's make use of the built-in 
> capabilities instead.
> if (log.level >= INFO) {
>    log.info("whatever " + method_that_builds_a_string + " : " + 
> some_variable);
> }
> etc.
> This would be a pretty huge thing to fix all-at-once so I suppose we'll have 
> to approach it incrementally. It's also something that, if we get them all 
> out of the code should be added to precommit failures. In the meantime, if 
> anyone who has the precommit chops could create a target that checked for 
> this it'd be a great help in tracking all of them down, then could be 
> incorporated in the regular precommit checks if/when they're all removed.
> Proposal:
> Use JFR or whatever to identify the egregious violations of this kind of 
> thing (I have a couple I've found) and change them to parameterized form (and 
> prove it works). Then see what we can do to move forward with removing them 
> all through the code base.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464563#comment-16464563
 ] 

Mark Miller commented on SOLR-11881:


bq. if we don't set the timeout on the HttpPost request by setting a request 
config

Cool - the tricky part is, if only one of the properties of the two is 
overridden on the client itself on the fly, we still want to pick up the 
default for the other one.

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" .
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10169) PeerSync will hit an NPE on no response errors when looking for fingerprint.

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464562#comment-16464562
 ] 

Erick Erickson commented on SOLR-10169:
---

I'm reluctant to start throwing more stuff at 7.3.1. If this was something that 
used to work in 7.2 and was broken in 7.3 it would be a different story. But 
this behavior has been around since forever. So my preference would be to just 
let it flow into 7.4.

I have no objection if someone else wants to backport to 7.3.1, but I don't 
have the motivation/time, especially as I'll be on vacation soon.

> PeerSync will hit an NPE on no response errors when looking for fingerprint.
> 
>
> Key: SOLR-10169
> URL: https://issues.apache.org/jira/browse/SOLR-10169
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.6, 7.0
>Reporter: Mark Miller
>Assignee: Erick Erickson
>Priority: Major
> Fix For: 7.4
>
> Attachments: SOLR-10169.patch, SOLR-10169.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12314) ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the solr.xml file

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464561#comment-16464561
 ] 

Varun Thacker commented on SOLR-12314:
--

Quick patch. I'll run some tests now. 

The other usages of HttpClientUtil.createDefaultRequestConfigBuilder() looked 
okay to me.

> ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the 
> solr.xml file
> -
>
> Key: SOLR-12314
> URL: https://issues.apache.org/jira/browse/SOLR-12314
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12314.patch
>
>
> In ConcurrentUpdateSolrClient we create an HttpPost Request which allows you 
> to set a request config. If the request config is not provided httpclient 
> will use the default request config. 
>  
> {code:java}
> org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
> HttpClientUtil.createDefaultRequestConfigBuilder();
> if (soTimeout != null) {
>   requestConfigBuilder.setSocketTimeout(soTimeout);
> }
> if (connectionTimeout != null) {
>   requestConfigBuilder.setConnectTimeout(connectionTimeout);
> }
> method.setConfig(requestConfigBuilder.build());{code}
> While creating the httpclient object we ensure that the default request is 
> set with the properties we care about.  This happens in 
> HttpClientUtils#setupBuilder
> {code:java}
> RequestConfig requestConfig = requestConfigBuilder.build();
> HttpClientBuilder retBuilder = 
> builder.setDefaultRequestConfig(requestConfig);{code}
> So there is no need to set a per request config 
>  
> Here is where the httpclient picks the request config is provided on the 
> request itself : 
> [https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L168]
>  
> And if it's not provided it uses the default here : 
> https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L148



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12314) ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the solr.xml file

[Varun Thacker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker updated SOLR-12314:
-
Attachment: SOLR-12314.patch

> ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the 
> solr.xml file
> -
>
> Key: SOLR-12314
> URL: https://issues.apache.org/jira/browse/SOLR-12314
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-12314.patch
>
>
> In ConcurrentUpdateSolrClient we create an HttpPost Request which allows you 
> to set a request config. If the request config is not provided httpclient 
> will use the default request config. 
>  
> {code:java}
> org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
> HttpClientUtil.createDefaultRequestConfigBuilder();
> if (soTimeout != null) {
>   requestConfigBuilder.setSocketTimeout(soTimeout);
> }
> if (connectionTimeout != null) {
>   requestConfigBuilder.setConnectTimeout(connectionTimeout);
> }
> method.setConfig(requestConfigBuilder.build());{code}
> While creating the httpclient object we ensure that the default request is 
> set with the properties we care about.  This happens in 
> HttpClientUtils#setupBuilder
> {code:java}
> RequestConfig requestConfig = requestConfigBuilder.build();
> HttpClientBuilder retBuilder = 
> builder.setDefaultRequestConfig(requestConfig);{code}
> So there is no need to set a per request config 
>  
> Here is where the httpclient picks the request config is provided on the 
> request itself : 
> [https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L168]
>  
> And if it's not provided it uses the default here : 
> https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L148



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12314) ConcurrentUpdateSolrClient doesn't respect the timeout's defined in the solr.xml file

[Varun Thacker (JIRA)]

Varun Thacker created SOLR-12314:


 Summary: ConcurrentUpdateSolrClient doesn't respect the timeout's 
defined in the solr.xml file
 Key: SOLR-12314
 URL: https://issues.apache.org/jira/browse/SOLR-12314
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
Assignee: Varun Thacker


In ConcurrentUpdateSolrClient we create an HttpPost Request which allows you to 
set a request config. If the request config is not provided httpclient will use 
the default request config. 

 
{code:java}
org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
HttpClientUtil.createDefaultRequestConfigBuilder();
if (soTimeout != null) {
  requestConfigBuilder.setSocketTimeout(soTimeout);
}
if (connectionTimeout != null) {
  requestConfigBuilder.setConnectTimeout(connectionTimeout);
}

method.setConfig(requestConfigBuilder.build());{code}
While creating the httpclient object we ensure that the default request is set 
with the properties we care about.  This happens in HttpClientUtils#setupBuilder
{code:java}
RequestConfig requestConfig = requestConfigBuilder.build();

HttpClientBuilder retBuilder = 
builder.setDefaultRequestConfig(requestConfig);{code}
So there is no need to set a per request config 

 

Here is where the httpclient picks the request config is provided on the 
request itself : 
[https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L168]

 

And if it's not provided it uses the default here : 
https://github.com/apache/httpcomponents-client/blob/4.5.3/httpclient/src/main/java/org/apache/http/impl/client/InternalHttpClient.java#L148



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464560#comment-16464560
 ] 

Erick Erickson commented on LUCENE-7976:


Shawn:

Much of the point of this work is exactly to change the behavior. The old code 
did this massive merge into a single segment for forceMerge which made it easy 
to shoot oneself in the foot. forceMerge, expungeDeletes, and IndexUpgraderTool 
have the same problem: they produce very large segments that then aren't merged 
away unless they have < maxSegmentSizeMB/2 "live" docs. There's a blog about 
that... 
https://lucidworks.com/2017/10/13/segment-merging-deleted-documents-optimize-may-bad/

I thought a bit about a new policy, but the motivation here is to alter the 
behavior of producing very large segments so I decided to approach it by 
altering TMP, rather than creating a new policy and deprecating TMP.

FWIW.

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12297) Create a good SolrClient for SolrCloud paving the way for async requests, HTTP2, multiplexing, and the latest & greatest Jetty features.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464559#comment-16464559
 ] 

Mark Miller commented on SOLR-12297:


I've grown even more confident we should move away from Apache's HttpClient.
 * It's been slow to support HTTP2 (support is in 5 beta currently).
 * There is a separate sub project called AsyncHttpClient for async support.
 * HTTP2 is a different API than HTTP1/1.1
 * I've never been a fan of the API itself.
 * The project has fairly low activity compared to Jetty.

For Jetty HttpClient
 * There is a single client for blocking and async support. That client also 
works for HTTP2 (although there is a lower level client as well).
 * The same project releases both client and server, testing them together, 
building them together.
 * Anything Jetty supports or fixes will flow much faster to their own client.
 * There API rewrite is much nicer IMO than what Apache HttpClient has been.
 * The Jetty team is very responsive to our project.

> Create a good SolrClient for SolrCloud paving the way for async requests, 
> HTTP2, multiplexing, and the latest & greatest Jetty features.
> 
>
> Key: SOLR-12297
> URL: https://issues.apache.org/jira/browse/SOLR-12297
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Priority: Major
>
> Blocking or async support as well as HTTP2 compatible with multiplexing.
> Once it supports enough and is stable, replace internal usage, allowing 
> async, and eventually move to HTTP2 connector and allow multiplexing. Could 
> support HTTP1.1 and HTTP2 on different ports depending on state of the world 
> then.
> The goal of the client itself is to work against HTTP1.1 or HTTP2 with 
> minimal or no code path differences and the same for async requests (should 
> initially work for both 1.1 and 2 and share majority of code).
> The client should also be able to replace HttpSolrClient and plug into the 
> other clients the same way.
> I doubt it would make sense to keep ConcurrentUpdateSolrClient eventually 
> though.
> I evaluated some clients and while there are a few options, I went with 
> Jetty's HttpClient. It's more mature than Apache HttpClient's support (in 5 
> beta) and we would have to update to a new API for Apache HttpClient anyway.
> Meanwhile, the Jetty guys have been very supportive of helping Solr with any 
> issues and I like having the client and server from the same project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464557#comment-16464557
 ] 

Varun Thacker commented on SOLR-11881:
--

{quote} I think if timeouts are null, we need to try and pull them from the 
httpclient?
{quote}
I followed the code and if we don't set the timeout on the HttpPost request by 
setting a request config , it will use the default request config. In our case 
we set the default request config while creating the httpclient in 
HttpClientUtil#setupBuilder so it will use the values defined in the solr.xml 
file . I'll file a separate Jira right now
{code:java}
RequestConfig requestConfig = requestConfigBuilder.build();

HttpClientBuilder retBuilder = 
builder.setDefaultRequestConfig(requestConfig);{code}

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" .
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12293) Updates need to use their own connection pool to maintain connection reuse and prevent spurious recoveries.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464545#comment-16464545
 ] 

ASF subversion and git services commented on SOLR-12293:


Commit 3a2572db793b47a6648fae8288a5c8815b689cd1 in lucene-solr's branch 
refs/heads/master from [~mark.mil...@oblivion.ch]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=3a2572d ]

SOLR-12293: Updates need to use their own connection pool to maintain 
connection reuse and prevent spurious recoveries.


> Updates need to use their own connection pool to maintain connection reuse 
> and prevent spurious recoveries.
> ---
>
> Key: SOLR-12293
> URL: https://issues.apache.org/jira/browse/SOLR-12293
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Attachments: SOLR-12293.patch
>
>
> Currently the pool is shared too broadly - for example during replication we 
> don't guarantee we read the full streams when downloading index files and we 
> don't necessarily want to, emptying the stream for a huge file due to error 
> or abort is too expensive. We can't have these connections pollute the update 
> connection pool.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464539#comment-16464539
 ] 

Mark Miller commented on SOLR-11881:


[~varunthacker], when we update the httpclient we could no longer set things we 
did after construction - but our client api let you change these settings on 
the fly (I see those methods are deprecated now - good). So this was to not 
break that. I think if timeouts are null, we need to try and pull them from the 
httpclient?

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" .
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464534#comment-16464534
 ] 

Shawn Heisey commented on SOLR-11453:
-

The rollover size value in the default config used to be 4MB, so 32MB is a huge 
improvement!  I agree, it's a low value.  But we can't make it TOO big, or Solr 
may have trouble when somebody begins using it on a very small system.  (Think 
raspberry pi, or a super small container/VM).  For the same reason, the default 
heap size is only 512MB, which means that almost all users have to eventually 
adjust their config.

When I deploy Solr for real usage, I change the rollover size to 4GB (on older 
versions where it's log4j.properties).  The systems I run it on have terabytes 
of disk space, and I want to be sure that I WILL have something to look at 
(without checking lots of log files) when somebody tells me about a problem 
they had two or three days ago.

My dev server is running an older version (6.6.3).  The production servers are 
running even older versions.  The current logfile on dev is 720MB and covers 
these times:

2018-05-02 15:26:59.556
2018-05-05 00:22:35.226

I haven't modified the logging config much, so that should be fairly typical 
for a server matching the size and activity level.

A fair amount of effort has been expended to reduce how much Solr logs in an 
out-of-the-box configuration.  When starting the Solr example on the most 
current release (7.3.0) with one index that has nothing in it, the logfile gets 
to 6KB.  Here's the contents:

{noformat}
2018-05-05 00:21:48.416 INFO  (main) [   ] o.e.j.s.Server 
jetty-9.4.8.v20171121, build timestamp: 2017-11-21T21:27:37Z, git hash: 
82b8fb23f757335bb3329d540ce37a2a2615f0a8
2018-05-05 00:21:49.133 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter  ___  
_   Welcome to Apache Solr™ version 7.3.0
2018-05-05 00:21:49.133 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter / __| 
___| |_ _   Starting in standalone mode on port 8983
2018-05-05 00:21:49.133 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter \__ \/ _ 
\ | '_|  Install dir: C:\Users\sheisey\Downloads\solr-7.3.0
2018-05-05 00:21:49.148 INFO  (main) [   ] o.a.s.s.SolrDispatchFilter 
|___/\___/_|_|Start time: 2018-05-05T00:21:49.135Z
2018-05-05 00:21:49.150 INFO  (main) [   ] o.a.s.u.StartupLoggingUtils Property 
solr.log.muteconsole given. Muting ConsoleAppender named CONSOLE
2018-05-05 00:21:49.238 INFO  (main) [   ] o.a.s.c.SolrResourceLoader Using 
system property solr.solr.home: 
C:\Users\sheisey\Downloads\solr-7.3.0\server\solr
2018-05-05 00:21:49.245 INFO  (main) [   ] o.a.s.c.SolrXmlConfig Loading 
container configuration from 
C:\Users\sheisey\Downloads\solr-7.3.0\server\solr\solr.xml
2018-05-05 00:21:50.197 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Found 
1 core definitions underneath C:\Users\sheisey\Downloads\solr-7.3.0\server\solr
2018-05-05 00:21:50.198 INFO  (main) [   ] o.a.s.c.CorePropertiesLocator Cores 
are: [foo]
2018-05-05 00:21:50.205 INFO  (coreLoadExecutor-6-thread-1) [   ] 
o.a.s.c.TransientSolrCoreCacheDefault Allocating transient cache for 2147483647 
transient cores
2018-05-05 00:21:50.323 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.SolrResourceLoader [foo] Added 57 libs to classloader, from paths: 
[/C:/Users/sheisey/Downloads/solr-7.3.0/contrib/clustering/lib, 
/C:/Users/sheisey/Downloads/solr-7.3.0/contrib/extraction/lib, 
/C:/Users/sheisey/Downloads/solr-7.3.0/contrib/langid/lib, 
/C:/Users/sheisey/Downloads/solr-7.3.0/contrib/velocity/lib, 
/C:/Users/sheisey/Downloads/solr-7.3.0/dist]
2018-05-05 00:21:53.197 INFO  (main) [   ] o.e.j.s.Server Started @6060ms
2018-05-05 00:21:53.898 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.SolrConfig Using Lucene MatchVersion: 7.3.0
2018-05-05 00:21:54.191 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.s.IndexSchema [foo] Schema name=default-config
2018-05-05 00:21:54.630 WARN  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.SolrResourceLoader Solr loaded a deprecated plugin/analysis class 
[solr.SynonymFilterFactory]. Please consult documentation how to replace it 
accordingly.
2018-05-05 00:21:55.357 INFO  (qtp818403870-15) [   ] o.a.s.s.HttpSolrCall 
[admin] webapp=null path=/admin/info/system params={wt=json} status=0 QTime=51
2018-05-05 00:21:55.371 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.s.IndexSchema Loaded schema default-config/1.6 with uniqueid field id
2018-05-05 00:21:55.438 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.CoreContainer Creating SolrCore 'foo' using configuration from 
instancedir C:\Users\sheisey\Downloads\solr-7.3.0\server\solr\foo, trusted=true
2018-05-05 00:21:55.485 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.SolrCore solr.RecoveryStrategy.Builder
2018-05-05 00:21:55.492 INFO  (coreLoadExecutor-6-thread-1) [   x:foo] 
o.a.s.c.SolrCore [[foo] ] Opening new SolrCore at 

[jira] [Commented] (SOLR-12313) TestInjection#waitForInSyncWithLeader needs improvement.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464535#comment-16464535
 ] 

Mark Miller commented on SOLR-12313:


Yeah, I'm seeing BasicDistributedZkTest runs that are like 500s but still pass 
because of this. We should reduce our request timeouts for tests to catch weird 
stuff like this.

> TestInjection#waitForInSyncWithLeader needs improvement.
> 
>
> Key: SOLR-12313
> URL: https://issues.apache.org/jira/browse/SOLR-12313
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Priority: Major
>
> This really should have some doc for why it would be used.
> I also think it causes BasicDistributedZkTest to take forever for sometimes 
> and perhaps other tests?
> I think checking for uncommitted data is probably a race condition and should 
> be removed.
> Checking index versions should follow the rules that replication does - if 
> the slave is higher than the leader, it's in sync, being equal is not 
> required. If it's expected for a test it should be a specific test that 
> fails. This just introduces massive delays.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464533#comment-16464533
 ] 

Tomás Fernández Löbbe commented on SOLR-11881:
--

I believe my current patch breaks the "minRf" behavior. I'll take a look and 
add a test for that

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" .
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk1.8.0_144) - Build # 7301 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/7301/
Java: 64bit/jdk1.8.0_144 -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC

14 tests failed.
FAILED:  org.apache.solr.handler.component.SpellCheckComponentTest.testCollate

Error Message:
Directory 
(MMapDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.handler.component.SpellCheckComponentTest_68B9CD720569970F-002\init-core-data-001\spellchecker1
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d15ecb4) still has 
pending deleted files; cannot initialize IndexWriter

Stack Trace:
java.lang.IllegalArgumentException: Directory 
(MMapDirectory@C:\Users\jenkins\workspace\Lucene-Solr-master-Windows\solr\build\solr-core\test\J1\temp\solr.handler.component.SpellCheckComponentTest_68B9CD720569970F-002\init-core-data-001\spellchecker1
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@1d15ecb4) still has 
pending deleted files; cannot initialize IndexWriter
at 
__randomizedtesting.SeedInfo.seed([68B9CD720569970F:A2989446AF91AA64]:0)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:699)
at 
org.apache.lucene.search.spell.SpellChecker.clearIndex(SpellChecker.java:455)
at 
org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:87)
at 
org.apache.solr.handler.component.SpellCheckComponent.prepare(SpellCheckComponent.java:128)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2510)
at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:982)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:951)
at 
org.apache.solr.handler.component.SpellCheckComponentTest.testCollate(SpellCheckComponentTest.java:171)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 

[JENKINS] Lucene-Solr-repro - Build # 579 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/579/

[...truncated 39 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-Tests-master/2517/consoleText

[repro] Revision: ab11867364fb93305d3174e99d869c070c57023c

[repro] Repro line:  ant test  -Dtestcase=TestTriggerIntegration 
-Dtests.method=testTriggerThrottling -Dtests.seed=2EFA8899AF0488A9 
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.locale=sr-Latn-ME 
-Dtests.timezone=Asia/Calcutta -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
296201055f24f01e1610f2fb87aba7fa90b9dda1
[repro] git fetch

[...truncated 2 lines...]
[repro] git checkout ab11867364fb93305d3174e99d869c070c57023c

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestTriggerIntegration
[repro] ant compile-test

[...truncated 3298 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestTriggerIntegration" -Dtests.showOutput=onerror  
-Dtests.seed=2EFA8899AF0488A9 -Dtests.multiplier=2 -Dtests.slow=true 
-Dtests.locale=sr-Latn-ME -Dtests.timezone=Asia/Calcutta -Dtests.asserts=true 
-Dtests.file.encoding=US-ASCII

[...truncated 5412 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: 
org.apache.solr.cloud.autoscaling.sim.TestTriggerIntegration
[repro] git checkout 296201055f24f01e1610f2fb87aba7fa90b9dda1

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12297) Create a good SolrClient for SolrCloud paving the way for async requests, HTTP2, multiplexing, and the latest & greatest Jetty features.

[Mark Miller (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mark Miller updated SOLR-12297:
---
Summary: Create a good SolrClient for SolrCloud paving the way for async 
requests, HTTP2, multiplexing, and the latest & greatest Jetty features.  (was: 
Create a good SolrClient for SolrCloud.)

> Create a good SolrClient for SolrCloud paving the way for async requests, 
> HTTP2, multiplexing, and the latest & greatest Jetty features.
> 
>
> Key: SOLR-12297
> URL: https://issues.apache.org/jira/browse/SOLR-12297
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Priority: Major
>
> Blocking or async support as well as HTTP2 compatible with multiplexing.
> Once it supports enough and is stable, replace internal usage, allowing 
> async, and eventually move to HTTP2 connector and allow multiplexing. Could 
> support HTTP1.1 and HTTP2 on different ports depending on state of the world 
> then.
> The goal of the client itself is to work against HTTP1.1 or HTTP2 with 
> minimal or no code path differences and the same for async requests (should 
> initially work for both 1.1 and 2 and share majority of code).
> The client should also be able to replace HttpSolrClient and plug into the 
> other clients the same way.
> I doubt it would make sense to keep ConcurrentUpdateSolrClient eventually 
> though.
> I evaluated some clients and while there are a few options, I went with 
> Jetty's HttpClient. It's more mature than Apache HttpClient's support (in 5 
> beta) and we would have to update to a new API for Apache HttpClient anyway.
> Meanwhile, the Jetty guys have been very supportive of helping Solr with any 
> issues and I like having the client and server from the same project.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464502#comment-16464502
 ] 

Varun Thacker commented on SOLR-11881:
--

So recently I've been seeing this problem in this form:

- The replica get's a ReadPendingException from Jetty 
{code:java}
date time WARN [qtp768306356-580185] ? (:) - 
java.nio.channels.ReadPendingException: null
at org.eclipse.jetty.io.FillInterest.register(FillInterest.java:58) 
~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]
at 
org.eclipse.jetty.io.AbstractEndPoint.fillInterested(AbstractEndPoint.java:353) 
~[jetty-io-9.4.8.v20171121.jar:9.4.8.v20171121]{code}
 * The leader keeps waiting till socket timeout and then get's a socket timeout 
exception and put's the replica into recovery

So I took Tomás latest patch and added SocketTimeoutException to the 
{{isRetriableException}} check. 

Q: What all exceptions should we retry on? Currently in the patch we have 
SocketException / NoHttpResponseException

 

Once I added SocketTimeoutException as a retriable exception , I then set the 
socket timeout to 100ms and sent updates to manually test if Solr's retrying 
correctly . To my surprise I was never able to hit a socket timeout exception . 
After some debugging here's why

In ConcurrentUpdateSolrClient we do this
{code:java}
org.apache.http.client.config.RequestConfig.Builder requestConfigBuilder = 
HttpClientUtil.createDefaultRequestConfigBuilder();
if (soTimeout != null) {
  requestConfigBuilder.setSocketTimeout(soTimeout);
}
if (connectionTimeout != null) {
  requestConfigBuilder.setConnectTimeout(connectionTimeout);
}

method.setConfig(requestConfigBuilder.build());{code}
So createDefaultRequestConfigBuilder doesn't respect the timeout set in 
solr.xml and uses a default of 60 seconds.

I debugged the code and if we simply remove these lines then the http-client 
will use the default requestConfig which Solr creates with the settings 
specified from the solr.xml file. 

Mark : Do you remember the motivation for overriding the defaults from update 
shard handlers httpclient and explicitly specifying a RequestConfig in CUSC? 
Happy to track this in a separate Jira 

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> 

[jira] [Commented] (SOLR-12313) TestInjection#waitForInSyncWithLeader needs improvement.

[Mark Miller (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464501#comment-16464501
 ] 

Mark Miller commented on SOLR-12313:


I have to dig into this more to understand why this was sometimes slowing 
things up so much or not passing when it should (main reason I saw for that was 
when replication things things are in sync, but this check does not). The 
description is just a dump from weird stuff I was seeing in the middle of the 
night.

Given that it's injected for all commits in every test randomly it looks, we 
want to make sure it's not slowing up tests in certain cases when it should 
not, or taking the full timeout for legit cases.

> TestInjection#waitForInSyncWithLeader needs improvement.
> 
>
> Key: SOLR-12313
> URL: https://issues.apache.org/jira/browse/SOLR-12313
> Project: Solr
>  Issue Type: Test
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Priority: Major
>
> This really should have some doc for why it would be used.
> I also think it causes BasicDistributedZkTest to take forever for sometimes 
> and perhaps other tests?
> I think checking for uncommitted data is probably a race condition and should 
> be removed.
> Checking index versions should follow the rules that replication does - if 
> the slave is higher than the leader, it's in sync, being equal is not 
> required. If it's expected for a test it should be a specific test that 
> fails. This just introduces massive delays.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464499#comment-16464499
 ] 

Remko Popma commented on SOLR-11453:


[~elyograg] Thanks for showing the full config. I notice that rollover occurs 
after the log file exceeds 32MB. That seems very small to me. Not sure how much 
logging you expect, but you may end up with a limited number of small log files 
and lose crucial troubleshooting info.

Also, you may be interested in the [custom delete on rollover 
action|https://logging.apache.org/log4j/2.x/manual/appenders.html#CustomDeleteOnRollover]
 that is more powerful than what can be achieved with  
{{}}.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12290) Do not close any servlet streams and improve our servlet stream closing prevention code for users and devs.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464484#comment-16464484
 ] 

ASF subversion and git services commented on SOLR-12290:


Commit 296201055f24f01e1610f2fb87aba7fa90b9dda1 in lucene-solr's branch 
refs/heads/master from [~mark.mil...@oblivion.ch]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=2962010 ]

SOLR-12290: Do not close any servlet streams and improve our servlet stream 
closing prevention code for users and devs.


> Do not close any servlet streams and improve our servlet stream closing 
> prevention code for users and devs.
> ---
>
> Key: SOLR-12290
> URL: https://issues.apache.org/jira/browse/SOLR-12290
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-12290.patch, SOLR-12290.patch, SOLR-12290.patch, 
> SOLR-12290.patch
>
>
> Original Summary:
> When you fetch a file for replication we close the request output stream 
> after writing the file which ruins the connection for reuse.
> We can't close response output streams, we need to reuse these connections. 
> If we do close them, clients are hit with connection problems when they try 
> and reuse the connection from their pool.
> New Summary:
> At some point the above was addressed during refactoring. We should remove 
> these neutered closes and review our close shield code.
> If you are here to track down why this is done:
> Connection reuse requires that we read all streams and do not close them - 
> instead the container itself must manage request and response streams. If we 
> allow them to be closed, not only do we lose some connection reuse, but we 
> can cause spurious client errors that can cause expensive recoveries for no 
> reason. The spec allows us to count on the container to manage streams. It's 
> our job simply to not close them and to always read them fully, from client 
> and server. 
> Java itself can help with always reading the streams fully up to some small 
> default amount of unread stream slack, but that is very dangerous to count 
> on, so we always manually eat up anything on the streams our normal logic 
> ends up not reading for whatever reason.
> We also cannot call abort without ruining the connection or sendError. These 
> should be options of very last resort (requiring a blood sacrifice) or when 
> shutting down.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12313) TestInjection#waitForInSyncWithLeader needs improvement.

[Mark Miller (JIRA)]

Mark Miller created SOLR-12313:
--

 Summary: TestInjection#waitForInSyncWithLeader needs improvement.
 Key: SOLR-12313
 URL: https://issues.apache.org/jira/browse/SOLR-12313
 Project: Solr
  Issue Type: Test
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Mark Miller


This really should have some doc for why it would be used.
I also think it causes BasicDistributedZkTest to take forever for sometimes and 
perhaps other tests?

I think checking for uncommitted data is probably a race condition and should 
be removed.

Checking index versions should follow the rules that replication does - if the 
slave is higher than the leader, it's in sync, being equal is not required. If 
it's expected for a test it should be a specific test that fails. This just 
introduces massive delays.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464476#comment-16464476
 ] 

Shawn Heisey commented on SOLR-11453:
-

Should we keep the slow query logging at WARN?  Initial thought is that this is 
the correct level, but I'm curious what everyone else thinks.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464473#comment-16464473
 ] 

Shawn Heisey commented on SOLR-11453:
-

This is the log4j2 config that Solr 7.4 will have out of the box:

{code:xml}




  

  

  %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} 
%X{replica} %X{core}] %c{1.} %m%n

  


  

  %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} 
%X{replica} %X{core}] %c{1.} %m%n

  
  


  
  

  
  





  
  

  

{code}

I will attempt to write a combined config that accomplishes the logging.

[~varunthacker], my bias (for the first release where we have this) is to not 
have this separate logfile created by default, but to have a commented config 
section in log4j2.xml that would enable it.  For a later release (still need to 
decide whether that's 7.x or 8.0), after we are sure it's working right in the 
wild, we can enable it by default.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-11453) Create separate logger for slow requests

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464452#comment-16464452
 ] 

Remko Popma edited comment on SOLR-11453 at 5/4/18 10:22 PM:
-

It may be easier to discuss here rather than on the log4j mailing list, so here 
is one idea. 

Configure log4j so that a certain named logger is associated with a separate 
appender. In your application code, all logging that needs to end up in the 
"slow requests" log file should be done using the logger with that name. 

{noformat}


  
logs/app.log
logs/app-slow.log
  
  

  


  

  
  

  


  

  

{noformat}
In your application:
{code}
private final static Logger slowRequests = 
LogManager.getLogger("org.apache.solr.core.SolrCore.SlowRequest");
private final static Logger logger = LogManager.getLogger();

...
logger.trace("normal logging");
slowRequests.trace("a slow request just came in");
{code}

With the above configuration, all log messages sent to the {{slowRequests}} 
logger end up in both the log files. If you want to send the slow request log 
messages to the slow log file only, set {{additivity = "false"}} on the slow 
logger. See the 
[manual|https://logging.apache.org/log4j/2.x/manual/configuration.html#Additivity]
 for details.

{noformat}
...
  

...
{noformat}



was (Author: rem...@yahoo.com):
It may be easier to discuss here rather than on the log4j mailing list, so here 
is one idea. 

Configure log4j so that a certain named logger is associated with a separate 
appender. In your application code, all logging that needs to end up in the 
"slow requests" log file should be done using the logger with that name. 

{noformat}


  
logs/app.log
logs/app-slow.log
  
  

  


  

  
  

  


  

  

{noformat}
In your application:
{code}
private final static Logger slowRequests = 
LogManager.getLogger("org.apache.solr.core.SolrCore.SlowRequest");
private final static Logger logger = LogManager.getLogger();

...
logger.trace("normal logging");
slowRequests.trace("a slow request just came in");
{code}

With the above configuration, all log messages sent to the {{slowRequests}} 
logger end up in both the log files. If you want to send the slow request log 
messages to the slow log file only, set {{additivity = "false"}} on the slow 
logger. See the 
[manual|https://logging.apache.org/log4j/2.x/manual/configuration.html#Additivity]
 for details.

{noformat}
...
  

...
{noformat}


> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk1.8.0_162) - Build # 21957 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/21957/
Java: 64bit/jdk1.8.0_162 -XX:-UseCompressedOops -XX:+UseG1GC

2 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([4DDEF31DDDF52FFC]:0)


FAILED:  org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction.testExecute

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([4DDEF31DDDF52FFC]:0)




Build Log:
[...truncated 14865 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.sim.TestExecutePlanAction
   [junit4]   2> 244489 INFO  
(SUITE-TestExecutePlanAction-seed#[4DDEF31DDDF52FFC]-worker) [] 
o.a.s.SolrTestCaseJ4 SecureRandom sanity checks: 
test.solr.allowed.securerandom=null & java.security.egd=file:/dev/./urandom
   [junit4]   2> Creating dataDir: 
/home/jenkins/workspace/Lucene-Solr-master-Linux/solr/build/solr-core/test/J0/temp/solr.cloud.autoscaling.sim.TestExecutePlanAction_4DDEF31DDDF52FFC-001/init-core-data-001
   [junit4]   2> 244506 WARN  
(SUITE-TestExecutePlanAction-seed#[4DDEF31DDDF52FFC]-worker) [] 
o.a.s.SolrTestCaseJ4 startTrackingSearchers: numOpens=10 numCloses=10
   [junit4]   2> 244506 INFO  
(SUITE-TestExecutePlanAction-seed#[4DDEF31DDDF52FFC]-worker) [] 
o.a.s.SolrTestCaseJ4 Using PointFields (NUMERIC_POINTS_SYSPROP=true) 
w/NUMERIC_DOCVALUES_SYSPROP=true
   [junit4]   2> 244508 INFO  
(SUITE-TestExecutePlanAction-seed#[4DDEF31DDDF52FFC]-worker) [] 
o.a.s.SolrTestCaseJ4 Randomized ssl (true) and clientAuth (false) via: 
@org.apache.solr.util.RandomizeSSL(reason=, ssl=NaN, value=NaN, clientAuth=NaN)
   [junit4]   2> 244517 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Adding .auto_add_replicas and 
.scheduled_maintenance triggers
   [junit4]   2> 244518 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode 
version 0
   [junit4]   2> 244519 INFO  
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.SolrTestCaseJ4 ###Starting testIntegration
   [junit4]   2> 244522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Current znodeVersion 0, lastZnodeVersion -1
   [junit4]   2> 244522 DEBUG (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Processed trigger updates upto znodeVersion 0
   [junit4]   2> 244523 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.OverseerTriggerThread Refreshing /autoscaling.json with znode version 
1
   [junit4]   2> 244523 INFO  
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.s.SimCloudManager === Restarting OverseerTriggerThread and clearing 
object cache...
   [junit4]   2> 244523 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.ScheduledTriggers Shutting down scheduled thread pool executor now
   [junit4]   2> 244523 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.ScheduledTriggers Shutting down action executor now
   [junit4]   2> 244524 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for action executor
   [junit4]   2> 244524 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.ScheduledTriggers Awaiting termination for scheduled thread pool 
executor
   [junit4]   2> 244524 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.ScheduledTriggers ScheduledTriggers closed completely
   [junit4]   2> 244524 DEBUG 
(TEST-TestExecutePlanAction.testIntegration-seed#[4DDEF31DDDF52FFC]) [] 
o.a.s.c.a.OverseerTriggerThread OverseerTriggerThread has been closed explicitly
   [junit4]   2> 244524 WARN  (Simulated OverseerAutoScalingTriggerThread) [
] o.a.s.c.a.OverseerTriggerThread Exception initializing trigger 
.scheduled_maintenance, configuration ignored
   [junit4]   2> org.apache.lucene.store.AlreadyClosedException: 
ScheduledTriggers has been closed and cannot be used anymore
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.ScheduledTriggers.add(ScheduledTriggers.java:202)
 ~[java/:?]
   [junit4]   2>at 
org.apache.solr.cloud.autoscaling.OverseerTriggerThread.run(OverseerTriggerThread.java:234)
 [java/:?]
   [junit4]   2>at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162]
   [junit4]   2> 244524 WARN  (Simulated OverseerAutoScalingTriggerThread) [
] 

[jira] [Comment Edited] (SOLR-11453) Create separate logger for slow requests

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464452#comment-16464452
 ] 

Remko Popma edited comment on SOLR-11453 at 5/4/18 10:20 PM:
-

It may be easier to discuss here rather than on the log4j mailing list, so here 
is one idea. 

Configure log4j so that a certain named logger is associated with a separate 
appender. In your application code, all logging that needs to end up in the 
"slow requests" log file should be done using the logger with that name. 

{noformat}


  
logs/app.log
logs/app-slow.log
  
  

  


  

  
  

  


  

  

{noformat}
In your application:
{code}
private final static Logger slowRequests = 
LogManager.getLogger("org.apache.solr.core.SolrCore.SlowRequest");
private final static Logger logger = LogManager.getLogger();

...
logger.trace("normal logging");
slowRequests.trace("a slow request just came in");
{code}

With the above configuration, all log messages sent to the {{slowRequests}} 
logger end up in both the log files. If you want to send the slow request log 
messages to the slow log file only, set {{additivity = "false"}} on the slow 
logger. See the 
[manual|https://logging.apache.org/log4j/2.x/manual/configuration.html#Additivity]
 for details.

{noformat}
...
  

...
{noformat}



was (Author: rem...@yahoo.com):
It may be easier to discuss here rather than on the log4j mailing list, so here 
is one idea. 

Configure log4j so that a certain named logger is associated with a separate 
appender. In your application code, all logging that needs to end up in the 
"slow requests" log file should be done using the logger with that name. 

{noformat}


  
logs/app.log
logs/app-slow.log
  
  

  


  

  
  

  


  

  

{noformat}
In your application:
{code}
private final static Logger slowRequests = 
LogManager.getLogger("org.apache.solr.core.SolrCore.SlowRequest");
private final static Logger logger = LogManager.getLogger();

...
logger.trace("normal logging");
slowRequests.trace("a slow request just came in");
{code}

With the above configuration, all log messages sent to the {{slowRequests}} 
logger end up in both the log files. If you want to send the slow request log 
messages to the slow log file only, set {{additivity = "false"}} on the slow 
logger. See the 
[manual|https://logging.apache.org/log4j/2.x/manual/configuration.html#Additivity]
 for details.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Remko Popma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464452#comment-16464452
 ] 

Remko Popma commented on SOLR-11453:


It may be easier to discuss here rather than on the log4j mailing list, so here 
is one idea. 

Configure log4j so that a certain named logger is associated with a separate 
appender. In your application code, all logging that needs to end up in the 
"slow requests" log file should be done using the logger with that name. 

{noformat}


  
logs/app.log
logs/app-slow.log
  
  

  


  

  
  

  


  

  

{noformat}
In your application:
{code}
private final static Logger slowRequests = 
LogManager.getLogger("org.apache.solr.core.SolrCore.SlowRequest");
private final static Logger logger = LogManager.getLogger();

...
logger.trace("normal logging");
slowRequests.trace("a slow request just came in");
{code}

With the above configuration, all log messages sent to the {{slowRequests}} 
logger end up in both the log files. If you want to send the slow request log 
messages to the slow log file only, set {{additivity = "false"}} on the slow 
logger. See the 
[manual|https://logging.apache.org/log4j/2.x/manual/configuration.html#Additivity]
 for details.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12292) Make it easier to configure Solr with CORS

[Michael Braun (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464451#comment-16464451
 ] 

Michael Braun commented on SOLR-12292:
--

>From the Jetty CORS filter docs at 
>https://www.eclipse.org/jetty/documentation/9.4.x/cross-origin-filter.html , 
>it supports restricting on HTTP method, so would restricting to just GET be 
>sufficient? (Not sure of the exact requests needed). 

> Make it easier to configure Solr with CORS
> --
>
> Key: SOLR-12292
> URL: https://issues.apache.org/jira/browse/SOLR-12292
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Priority: Major
>
> While working on SOLR-8207 I wanted to collect info from other SolrCloud 
> nodes from the AdminUI. However this is blocked by 
> [CORS|https://en.wikipedia.org/wiki/Cross-origin_resource_sharing] policy. In 
> that Jira I instead did the fan-out on the Solr server side for the two 
> handler I needed.
> It would be nice if all nodes in a SolrCloud cluster could automatically 
> accept any other node as a legal origin, and make it easy for users to add 
> other origins by config.
> If we use the [Jetty CORS 
> filter|http://www.eclipse.org/jetty/documentation/9.4.9.v20180320/cross-origin-filter.html]
>  in web.xml, perhaps we could parse a env.var from solr.in.xx and inject into 
> the {{allowedOrigins}} property of that filter? There is also SOLR-6059 which 
> tries to implement CORS inside of Solr handlers and not in Jetty. Don't know 
> pros/cons of those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11865) Refactor QueryElevationComponent to prepare query subset matching

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464442#comment-16464442
 ] 

David Smiley commented on SOLR-11865:
-

Patch still in progress but want to mention some things.
 * New name {{useConfiguredOrderForElevations}}.  Documentation language: "When 
multiple docs are elevated, should their relative order be the order in the 
configuration file or should they be subject to whatever the sort criteria is? 
True by default."
 * Found a way to entirely skip using ElevationComparatorSource if the 
Elevation obj has no elevations (maybe just has exclusions)
 * I was looking in detail at ElevationComparatorSource that led me to some 
observations that I'd like your input on:
 ** BytesRef[] termValues is the "value half of the map".  It's the BytesRef 
version of the ID values aligned with ordSet (doc ID) slots.  But the reader of 
these (docVal()) has to do additional work to look it up in 
elevation.priorities to get an int.  I think this could be replaced with an 
int[] populated with the pertinent int priorities when doSetNextReader is 
called (which is where ordSet & termValues is init'ed right now).  This int[] 
would be named simply priorities.
 ** Elevation.elevatedIds could be a Map that maps directly to 
the priority from the uniqueKey val (thus removing the need for a separate 
"priorities" map), and then in doSetNextReader we can iterate on the Map.Entry 
and needn't do another lookup.
 ** I wonder if the String IDs in Elevation, both elevated and excluded, ought 
to be BytesRefs to clarify that they are raw indexed form IDs?  (consider when 
uniqueKey is a long)  The current String form is suggestive that they are the 
surface form IDs, yet they aren't since they've already been mapped with 
FieldType.readableToIndexed.  Or alternatively keep the surface form IDs and 
translate them at a later time.  I think we might as well do them eagerly as it 
saves work during search, even if it's easy work, and again it clarifies the 
type.
 *** FieldType.readableToIndexed(String) ought to be deprecated in lieu of 
readableToIndexed(CharSequence, BytesRefBuilder).
 ** I guess it's debatable where to actually apply the key String => indexed 
form (String of BytesRef)... we're doing it in Elevation's constructor with a 
passed in UnaryOperator thingy but it could just as easily be done very late 
in, say, ElevationComparatorSource.doSetNextReader, or perhaps very early right 
after we read it from the XML. I suppose it's fine as-is.

> Refactor QueryElevationComponent to prepare query subset matching
> -
>
> Key: SOLR-11865
> URL: https://issues.apache.org/jira/browse/SOLR-11865
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SearchComponents - other
>Affects Versions: master (8.0)
>Reporter: Bruno Roustant
>Priority: Minor
>  Labels: QueryComponent
> Fix For: master (8.0)
>
> Attachments: 
> 0001-Refactor-QueryElevationComponent-to-introduce-Elevat.patch, 
> 0002-Refactor-QueryElevationComponent-after-review.patch, 
> 0003-Remove-exception-handlers-and-refactor-getBoostDocs.patch, 
> SOLR-11865.patch
>
>
> The goal is to prepare a second improvement to support query terms subset 
> matching or query elevation rules.
> Before that, we need to refactor the QueryElevationComponent. We make it 
> extendible. We introduce the ElevationProvider interface which will be 
> implemented later in a second patch to support subset matching. The current 
> full-query match policy becomes a default simple MapElevationProvider.
> - Add overridable methods to handle exceptions during the component 
> initialization.
> - Add overridable methods to provide the default values for config properties.
> - No functional change beyond refactoring.
> - Adapt unit test.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Review Request 66967: SolrCmdDistributor retries requestss from leader to followers

[Tomás Fernández Löbbe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66967/
---

(Updated May 4, 2018, 10:04 p.m.)


Review request for lucene.


Changes
---

Removed log4j change. Fixed bug that broke test


Repository: lucene-solr


Description
---

Still need to do some more testing. Won't be committing the log4j changes


Diffs (updated)
-

  solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java 08b397fbb2 
  
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
 d5e4194b15 
  
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
 224bd4b6ba 
  solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java 
72d39ff89b 
  solr/core/src/test/org/apache/solr/update/SolrCmdDistributorTest.java 
1699b0d70d 


Diff: https://reviews.apache.org/r/66967/diff/2/

Changes: https://reviews.apache.org/r/66967/diff/1-2/


Testing
---


Thanks,

Tomás Fernández Löbbe

Re: Review Request 66967: SolrCmdDistributor retries requestss from leader to followers

[Tomás Fernández Löbbe]

> On May 4, 2018, 9:11 p.m., Varun Thacker wrote:
> > solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java
> > Line 28 (original)
> > 
> >
> > We can use @LogLevel("org.apache.solr.update.SolrCmdDistributor=DEBUG) 
> > for this test to make it easier to check manually if the retires are 
> > happening or not . This would avoid changing the log4j2.xml file

Thanks! I didn't know that existed.


- Tomás


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66967/#review202485
---


On May 4, 2018, 7:49 p.m., Tomás Fernández Löbbe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66967/
> ---
> 
> (Updated May 4, 2018, 7:49 p.m.)
> 
> 
> Review request for lucene.
> 
> 
> Repository: lucene-solr
> 
> 
> Description
> ---
> 
> Still need to do some more testing. Won't be committing the log4j changes
> 
> 
> Diffs
> -
> 
>   solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java 
> 08b397fbb2 
>   
> solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
>  d5e4194b15 
>   
> solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
>  224bd4b6ba 
>   solr/core/src/test-files/log4j2.xml 7d0ebf7a49 
>   solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java 
> 72d39ff89b 
>   solr/core/src/test/org/apache/solr/update/SolrCmdDistributorTest.java 
> 1699b0d70d 
> 
> 
> Diff: https://reviews.apache.org/r/66967/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tomás Fernández Löbbe
> 
>

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464433#comment-16464433
 ] 

Shawn Heisey commented on SOLR-11453:
-

I have revived the thread on the log4j mailing list to learn how to do it with 
the upgraded version.

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464414#comment-16464414
 ] 

Shawn Heisey commented on LUCENE-7976:
--

General thoughts: I think the overall goals stated in the class-level javadoc 
for TieredMergePolicy are good.  Based on the discussion, minimizing deleted 
docs (whether there has been a forceMerge in the past or not) needs to be a 
more significant focus than it is now.


> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12292) Make it easier to configure Solr with CORS

[Alexandre Rafalovitch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464413#comment-16464413
 ] 

Alexandre Rafalovitch commented on SOLR-12292:
--

JSONP is read-only though. So, it exposes less than CORS.

IF CORS is open than any webpage can hit the localhost and possibly inject 
stuff, creating a local exploit. 

This _may_ be possible with our implementation of JSONP as well, but the risk 
surface is much smaller.

> Make it easier to configure Solr with CORS
> --
>
> Key: SOLR-12292
> URL: https://issues.apache.org/jira/browse/SOLR-12292
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Server
>Reporter: Jan Høydahl
>Priority: Major
>
> While working on SOLR-8207 I wanted to collect info from other SolrCloud 
> nodes from the AdminUI. However this is blocked by 
> [CORS|https://en.wikipedia.org/wiki/Cross-origin_resource_sharing] policy. In 
> that Jira I instead did the fan-out on the Solr server side for the two 
> handler I needed.
> It would be nice if all nodes in a SolrCloud cluster could automatically 
> accept any other node as a legal origin, and make it easy for users to add 
> other origins by config.
> If we use the [Jetty CORS 
> filter|http://www.eclipse.org/jetty/documentation/9.4.9.v20180320/cross-origin-filter.html]
>  in web.xml, perhaps we could parse a env.var from solr.in.xx and inject into 
> the {{allowedOrigins}} property of that filter? There is also SOLR-6059 which 
> tries to implement CORS inside of Solr handlers and not in Jetty. Don't know 
> pros/cons of those.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Review Request 66967: SolrCmdDistributor retries requestss from leader to followers

[Varun Thacker]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66967/#review202485
---




solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java
Line 28 (original)


We can use @LogLevel("org.apache.solr.update.SolrCmdDistributor=DEBUG) for 
this test to make it easier to check manually if the retires are happening or 
not . This would avoid changing the log4j2.xml file


- Varun Thacker


On May 4, 2018, 7:49 p.m., Tomás Fernández Löbbe wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/66967/
> ---
> 
> (Updated May 4, 2018, 7:49 p.m.)
> 
> 
> Review request for lucene.
> 
> 
> Repository: lucene-solr
> 
> 
> Description
> ---
> 
> Still need to do some more testing. Won't be committing the log4j changes
> 
> 
> Diffs
> -
> 
>   solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java 
> 08b397fbb2 
>   
> solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
>  d5e4194b15 
>   
> solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
>  224bd4b6ba 
>   solr/core/src/test-files/log4j2.xml 7d0ebf7a49 
>   solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java 
> 72d39ff89b 
>   solr/core/src/test/org/apache/solr/update/SolrCmdDistributorTest.java 
> 1699b0d70d 
> 
> 
> Diff: https://reviews.apache.org/r/66967/diff/1/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tomás Fernández Löbbe
> 
>

[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464399#comment-16464399
 ] 

Shawn Heisey commented on LUCENE-7976:
--

I have not been following the code, but I have been following the discussion.  
How much of the complexity is related to not surprising existing users?

I think a lot of concerns would disappear if the focus is implementing a new 
merge policy instead of attempting to fix/augment TMP.

[~mikemccand], was TMP a ground-up implementation, or an evolution from LBMP?  
When I find some time, I can look at the code.

I've been on a little bit of a "let's rewrite it from scratch!" kick lately 
with regards to my own code, so keep that  in mind when evaluating any bias I 
might show!


> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment with > 20% deleted documents would be merged or rewritten NO 
> > MATTER HOW LARGE. There are two cases,
> >> the segment has < 5G "live" docs. In that case it would be merged with 
> >> smaller segments to bring the resulting segment up to 5G. If no smaller 
> >> segments exist, it would just be rewritten
> >> The segment has > 5G "live" docs (the result of a forceMerge or optimize). 
> >> It would be rewritten into a single segment removing all deleted docs no 
> >> matter how big it is to start. The 100G example above would be rewritten 
> >> to an 80G segment for instance.
> Of course this would lead to potentially much more I/O which is why the 
> default would be the same behavior we see now. As it stands now, though, 
> there's no way to recover from an optimize/forceMerge except to re-index from 
> scratch. We routinely see 200G-300G Lucene indexes at this point "in the 
> wild" with 10s of  shards replicated 3 or more times. And that doesn't even 
> include having these over HDFS.
> Alternatives welcome! Something like the above seems minimally invasive. A 
> new merge policy is certainly an alternative.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464394#comment-16464394
 ] 

David Smiley commented on SOLR-12312:
-

While I look at this a bit more in my IDE, I wonder if we even need this buf at 
all.  Notice that {{fetchPackets}} accepts {{FastInputStream fis}} and that's 
what we read from.  FIS contains an accessible {{byte[]}} internally; it's 
partially why FIS exists in the first place (vs some generic InputStream).  
fetchPackets could be recoded to have a loop within the packet reading to use 
FIS's buffer directly.  That would not be an abuse; we're just sending data 
along to other places (to a file and some checksum calculator thingy). The net 
effect is one less buffer to allocate and copy data to.

That said, I so easily become guilty of scope creep and this is extra work with 
more intricacies than the fairly simple change we have on sizing the buf 
correctly.

> Replication's IndexFetcher buf size can be initialized smarter to not waste 
> RAM/GC
> --
>
> Key: SOLR-12312
> URL: https://issues.apache.org/jira/browse/SOLR-12312
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> IndexFetcher's constructor knows the size of the file it's going to transfer. 
>  As-such, it ought to initialize the "buf" field to no larger than this size. 
>  This has been shown to waste Java heap/GC in  an environment with lots of 
> cores of small indexes and thus small files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464372#comment-16464372
 ] 

David Smiley commented on SOLR-12312:
-

Latest patch looks good [~millerjeff0]; I like how you identified the re-use of 
an existing constant for this that the writer is using.  I'll probably reword a 
comment since I can be obsessive about trivialities.  I'll commit "soon".

> Replication's IndexFetcher buf size can be initialized smarter to not waste 
> RAM/GC
> --
>
> Key: SOLR-12312
> URL: https://issues.apache.org/jira/browse/SOLR-12312
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: David Smiley
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> IndexFetcher's constructor knows the size of the file it's going to transfer. 
>  As-such, it ought to initialize the "buf" field to no larger than this size. 
>  This has been shown to waste Java heap/GC in  an environment with lots of 
> cores of small indexes and thus small files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley reassigned SOLR-12312:
---

Assignee: David Smiley

> Replication's IndexFetcher buf size can be initialized smarter to not waste 
> RAM/GC
> --
>
> Key: SOLR-12312
> URL: https://issues.apache.org/jira/browse/SOLR-12312
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: replication (java)
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> IndexFetcher's constructor knows the size of the file it's going to transfer. 
>  As-such, it ought to initialize the "buf" field to no larger than this size. 
>  This has been shown to waste Java heap/GC in  an environment with lots of 
> cores of small indexes and thus small files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Solaris (64bit/jdk1.8.0) - Build # 1839 - Failure!

[Policeman Jenkins Server]

Error processing tokens: Error while parsing action 
'Text/ZeroOrMore/FirstOf/Token/DelimitedToken/DelimitedToken_Action3' at input 
position (line 79, pos 4):
)"}
   ^

java.lang.OutOfMemoryError: Java heap space

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #370: SOLR-12312: Change buf to not always use up 1...

[millerjeff0]

Github user millerjeff0 commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/370#discussion_r186210634
  
--- Diff: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java ---
@@ -1629,8 +1630,6 @@ private int fetchPackets(FastInputStream fis) throws 
Exception {
 LOG.warn("No content received for file: {}", fileName);
 return NO_CONTENT;
   }
-  if (buf.length < packetSize)
--- End diff --

Hmm..ok that makes sense as a safety mechanism. It's effectively done by 
settings to PACKET_SZ anyways.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] lucene-solr pull request #370: SOLR-12312: Change buf to not always use up 1...

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/370#discussion_r186205859
  
--- Diff: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java ---
@@ -1629,8 +1630,6 @@ private int fetchPackets(FastInputStream fis) throws 
Exception {
 LOG.warn("No content received for file: {}", fileName);
 return NO_CONTENT;
   }
-  if (buf.length < packetSize)
--- End diff --

I appreciate that this check may be needless now but it feels awfully 
presumptuous to me that this could never happen.  Like what if PACKET_SZ 
changes some day and someone is upgrading their Solr cluster with mixed 
versions at the same time and a larger packet is sent?  Even an assert feels to 
strict.  It could happen but it's not expected to.  So leave it and add a 
simple one-liner comment that in practice this won't happen but we adjust in 
case it does.


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12303) Subquery Doc transform doesn't inherit path from original request

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464339#comment-16464339
 ] 

Mikhail Khludnev commented on SOLR-12303:
-

This makes much sense, I'll tweak it a little before commit. Aren't there 
issues with SolrCloud? 

> Subquery Doc transform doesn't inherit path from original request
> -
>
> Key: SOLR-12303
> URL: https://issues.apache.org/jira/browse/SOLR-12303
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Munendra S N
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-12303.patch, SOLR-12303.patch
>
>
> {code:java}
> localhost:8983/solr/k_test/search?sort=score desc,uniqueId 
> desc=AND=json={!parent which=parent_field:true score=max}({!edismax 
> v=$origQuery})=false=uniqueId=score=_children_:[subquery]=uniqueId=false=parent_field&_children_.fl=uniqueId&_children_.fl=score&_children_.rows=3=false&_children_.q={!edismax
>  qf=parentId v=$row.uniqueId}=1
> {code}
> For this request, even though the path is */search*, the subquery request 
> would be fired on handler */select*.
> Subquery request should inherit the parent request handler and there should 
> be an option to override this behavior. (option to override is already 
> available by specifying *qt*)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8998) JSON Facet API child roll-ups

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464332#comment-16464332
 ] 

Mikhail Khludnev commented on SOLR-8998:


make sense [~ctargett] I'll fix it next week. 

> JSON Facet API child roll-ups
> -
>
> Key: SOLR-8998
> URL: https://issues.apache.org/jira/browse/SOLR-8998
> Project: Solr
>  Issue Type: New Feature
>  Components: Facet Module
>Reporter: Yonik Seeley
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-8998-api-doc.patch, SOLR-8998-doc.patch, 
> SOLR-8998.patch, SOLR-8998.patch, SOLR-8998.patch, SOLR_8998.patch, 
> SOLR_8998.patch, SOLR_8998.patch
>
>
> The JSON Facet API currently has the ability to map between parents and 
> children ( see http://yonik.com/solr-nested-objects/ )
> This issue is about adding a true rollup ability where parents would take on 
> derived values from their children.  The most important part (and the most 
> difficult part) will be the external API.
> [~mkhludnev] says
> {quote}
> The bottom line is to introduce {{uniqueBlock(\_root_)}} aggregation, which 
> is supposed to be faster than {{unique(\_root_)}} and purposed for blocked 
> index only. For now it it supports singlevalue string fields, docValues 
> usually make sense.   
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11881) Connection Reset Causing LIR

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-11881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464325#comment-16464325
 ] 

Tomás Fernández Löbbe commented on SOLR-11881:
--

Uploaded a new patch to https://reviews.apache.org/r/66967/

> Connection Reset Causing LIR
> 
>
> Key: SOLR-11881
> URL: https://issues.apache.org/jira/browse/SOLR-11881
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Varun Thacker
>Assignee: Varun Thacker
>Priority: Major
> Attachments: SOLR-11881-SolrCmdDistributor.patch, SOLR-11881.patch, 
> SOLR-11881.patch
>
>
> We can see that a connection reset is causing LIR.
> If a leader -> replica update get's a connection like this the leader will 
> initiate LIR
> {code:java}
> 2018-01-08 17:39:16.980 ERROR (qtp1010030701-468988) [c:collection s:shardX 
> r:core_node56 collection_shardX_replicaY] 
> o.a.s.u.p.DistributedUpdateProcessor Setting up to try to start recovery on 
> replica https://host08.domain:8985/solr/collection_shardX_replicaY/
> java.net.SocketException: Connection reset
> at java.net.SocketInputStream.read(SocketInputStream.java:210)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
> at 
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
> at 
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:543)
> at 
> org.apache.http.conn.ssl.SSLSocketFactory.connectSocket(SSLSocketFactory.java:409)
> at 
> org.apache.http.impl.conn.DefaultClientConnectionOperator.openConnection(DefaultClientConnectionOperator.java:177)
> at 
> org.apache.http.impl.conn.ManagedClientConnectionImpl.open(ManagedClientConnectionImpl.java:304)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.tryConnect(DefaultRequestDirector.java:611)
> at 
> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:446)
> at 
> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at 
> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.sendUpdateStream(ConcurrentUpdateSolrClient.java:312)
> at 
> org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrClient$Runner.run(ConcurrentUpdateSolrClient.java:185)
> at 
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:229)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> From https://issues.apache.org/jira/browse/SOLR-6931 Mark says "On a heavy 
> working SolrCloud cluster, even a rare response like this from a replica can 
> cause a recovery and heavy cluster disruption" .
> Looking at SOLR-6931 we added a http retry handler but we only retry on GET 
> requests. Updates are POST requests 
> {{ConcurrentUpdateSolrClient#sendUpdateStream}}
> Update requests between the leader and replica should be retry-able since 
> they have been versioned.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Review Request 66967: SolrCmdDistributor retries requestss from leader to followers

[Tomás Fernández Löbbe]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/66967/
---

Review request for lucene.


Repository: lucene-solr


Description
---

Still need to do some more testing. Won't be committing the log4j changes


Diffs
-

  solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java 08b397fbb2 
  
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
 d5e4194b15 
  
solr/core/src/java/org/apache/solr/update/processor/TimeRoutedAliasUpdateProcessor.java
 224bd4b6ba 
  solr/core/src/test-files/log4j2.xml 7d0ebf7a49 
  solr/core/src/test/org/apache/solr/update/MockStreamingSolrClients.java 
72d39ff89b 
  solr/core/src/test/org/apache/solr/update/SolrCmdDistributorTest.java 
1699b0d70d 


Diff: https://reviews.apache.org/r/66967/diff/1/


Testing
---


Thanks,

Tomás Fernández Löbbe

[GitHub] lucene-solr pull request #370: SOLR-12312: Change buf to not always use up 1...

[dsmiley]

Github user dsmiley commented on a diff in the pull request:

https://github.com/apache/lucene-solr/pull/370#discussion_r186196579
  
--- Diff: solr/core/src/java/org/apache/solr/handler/IndexFetcher.java ---
@@ -1549,6 +1549,8 @@ public ReplicationHandlerException(String message) {
   this.file = file;
   this.fileName = (String) fileDetails.get(NAME);
   this.size = (Long) fileDetails.get(SIZE);
+  //Max buf of 1 MB, but we should save memory if the file size is 
smaller
+  buf = this.size < 1048576 ? new byte[(int) this.size] : new 
byte[1024 * 1024];
--- End diff --

I observe 1048576 is 1024*1024.
Lets define a constant DEFAULT_BUF_SIZE = 1024 * 1024  (or perhaps call 
MAX_BUF_SIZE?) and then use this constant twice here.  And you can simply use 
`Math.min(this.size(), DEFAULT_BUF_SIZE)`.  No need for any comment; it's plain 
enough (IMO).


---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-12249) Grouping on a solr.TextField works in stand-alone but not in SolrCloud

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson reassigned SOLR-12249:
-

Assignee: Erick Erickson

> Grouping on a solr.TextField works in stand-alone but not in SolrCloud
> --
>
> Key: SOLR-12249
> URL: https://issues.apache.org/jira/browse/SOLR-12249
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.6.2, master (8.0)
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
>
> I didn't test this with master. Under the covers in stand-alone mode, the 
> "min" function is silently  substituted for the grouping, but that's not true 
> in SorlCloud mode. I broke this JIRA out separately to discuss whether it 
> _ever_ makes sense to group by a tokenized text field.
> Grouping by the min value in a field is at least consistent, but on a text 
> field I don't think it makes sense.
> I propose that we explicitly dis-allow this in both stand-alone and Cloud 
> mode, especially now that there's the SortableTextField.
> Comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (SOLR-12248) Grouping in SolrCloud fails if indexed="false" docValues="true" and stored="false"

[Erick Erickson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Erick Erickson reassigned SOLR-12248:
-

Assignee: Erick Erickson

> Grouping in SolrCloud fails if indexed="false" docValues="true" and 
> stored="false"
> --
>
> Key: SOLR-12248
> URL: https://issues.apache.org/jira/browse/SOLR-12248
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.6.2
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
>
> In SolrCloud _only_ (it works in stand-alone mode), a field defined as:
>  indexed="false"  docValues="true"  stored="false"  />
> will fail with the following error:
> java.lang.NullPointerException
> org.apache.solr.schema.BoolField.toExternal(BoolField.java:131)
> org.apache.solr.schema.BoolField.toObject(BoolField.java:142)
> org.apache.solr.schema.BoolField.toObject(BoolField.java:51)
> org.apache.solr.search.grouping.endresulttransformer.GroupedEndResultTransformer.transform(GroupedEndResultTransformer.java:72)
> org.apache.solr.handler.component.QueryComponent.groupedFinishStage(QueryComponent.java:830)
> org.apache.solr.handler.component.QueryComponent.finishStage(QueryComponent.java:793)
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:435)
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:173)
> .
> .
> curiously enough it succeeds with a field identically defined except for 
> stored="true"



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS-EA] Lucene-Solr-7.x-Linux (64bit/jdk-11-ea+5) - Build # 1848 - Failure!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/1848/
Java: 64bit/jdk-11-ea+5 -XX:+UseCompressedOops -XX:+UseParallelGC

All tests passed

Build Log:
[...truncated 15264 lines...]
   [junit4] JVM J0: stdout was not empty, see: 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/solr/build/solr-core/test/temp/junit4-J0-20180504_183411_9775008932648395691205.sysout
   [junit4] >>> JVM J0 emitted unexpected output (verbatim) 
   [junit4] java.lang.OutOfMemoryError: Java heap space
   [junit4] Dumping heap to 
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/heapdumps/java_pid5572.hprof ...
   [junit4] Heap dump file created [442918349 bytes in 0.612 secs]
   [junit4] <<< JVM J0: EOF 

[...truncated 9428 lines...]
BUILD FAILED
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/build.xml:633: The following 
error occurred while executing this line:
/home/jenkins/workspace/Lucene-Solr-7.x-Linux/build.xml:585: Some of the tests 
produced a heap dump, but did not fail. Maybe a suppressed OutOfMemoryError? 
Dumps created:
* java_pid5572.hprof

Total time: 64 minutes 29 seconds
Build step 'Invoke Ant' marked build as failure
Archiving artifacts
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
[WARNINGS] Skipping publisher since build result is FAILURE
Recording test results
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Email was triggered for: Failure - Any
Sending email for trigger: Failure - Any
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2
Setting 
ANT_1_8_2_HOME=/var/lib/jenkins/tools/hudson.tasks.Ant_AntInstallation/ANT_1.8.2

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-7.x - Build # 212 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-7.x/212/

No tests ran.

Build Log:
[...truncated 24220 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2194 links (1751 relative) to 2949 anchors in 228 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/package/changes

-dist-keys:
  [get] Getting: http://home.apache.org/keys/group/lucene.asc
  [get] To: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/package/KEYS

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/package/solr-7.4.0.tgz
 into 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-7.x/lucene/top-level-ivy-settings.xml

[jira] [Commented] (SOLR-11694) UIMA Contrib no longer works with Alchemy keys

[Alexandre Rafalovitch (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464292#comment-16464292
 ] 

Alexandre Rafalovitch commented on SOLR-11694:
--

I had a look at this.

We are shipping UIMA 2.3.1, which has been released in 2010

The latest version 3.0 (a big redesign) with latest stable version 2.10.x

Most of the examples (OpenCalais, etc) we have in the distribution are dead as 
per this and other JIRAs. And between OpenNLP and LTR we probably support a lot 
of previously-interesting functionality in other ways.

I think this whole contribution is a dead-end. Maybe we should pull it 
completely and save the 6Mb it takes? The connector source code is in the 
version repository if somebody wanted to bring it up-to-date, but I don't think 
the current state is useful for anybody.

> UIMA Contrib no longer works with Alchemy keys
> --
>
> Key: SOLR-11694
> URL: https://issues.apache.org/jira/browse/SOLR-11694
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: contrib - UIMA
>Reporter: Cassandra Targett
>Priority: Major
>
> A user on the [solr-user mailing list back in 
> June|https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201706.mbox/%3CCANsk%2BC_PvZJ38AQ2VfzKRYSQn6c8b33kGvaXxR3qNS3GQ4VUKA%40mail.gmail.com%3E]
>  brought up the fact that IBM has bought Alchemy and keys are no longer 
> available to use Solr's UIMA contrib.
> Someone recently made a [similar 
> comment|https://lucene.apache.org/solr/guide/7_1/uima-integration.html#comment_7174]
>  to the Solr Ref Guide page and asking for a patch.
> I know next to nothing about UIMA, but figured it's worth an issue to 
> determine what to do here. I think folks are saying it's no longer usable? Or 
> maybe only usable by people who already have keys (which will possibly expire 
> at some point)?
> Anyone have an idea what needs to be done here? It seems we should have some 
> kind of answer, but if it's no longer usable perhaps we should retire the 
> contrib.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12280) Ref-Guide: Add Digital Signal Processing documentation

[Joel Bernstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Bernstein updated SOLR-12280:
--
Description: 
This ticket will add a new section of documentation in the Math Expressions 
docs covering the Digital Signal Processing functions. The main areas of 
documentation coverage will include:
 * Dot product
 * Convolution
 * Cross-correlation
 * Find Delay
 * Auto-correlation
 * Fast Fourier Transform

  was:
This ticket will add a new section of documentation in the Math Expressions 
docs covering the Digital Signal Processing functions. The main areas of 
documentation coverage will include:
 * Convolution
 * Cross-correlation
 * Find Delay
 * Auto-correlation
 * Fast Fourier Transform


> Ref-Guide: Add Digital Signal Processing documentation
> --
>
> Key: SOLR-12280
> URL: https://issues.apache.org/jira/browse/SOLR-12280
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Joel Bernstein
>Assignee: Joel Bernstein
>Priority: Major
> Fix For: 7.4
>
>
> This ticket will add a new section of documentation in the Math Expressions 
> docs covering the Digital Signal Processing functions. The main areas of 
> documentation coverage will include:
>  * Dot product
>  * Convolution
>  * Cross-correlation
>  * Find Delay
>  * Auto-correlation
>  * Fast Fourier Transform



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-7.x - Build # 213 - Still unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-7.x/213/

2 tests failed.
FAILED:  org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR

Error Message:
Error from server at http://127.0.0.1:37240/solr: KeeperErrorCode = Session 
expired for /overseer/collection-queue-work/qnr-

Stack Trace:
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error 
from server at http://127.0.0.1:37240/solr: KeeperErrorCode = Session expired 
for /overseer/collection-queue-work/qnr-
at 
__randomizedtesting.SeedInfo.seed([2B3BC342F5795C90:71A3F9848BF93B77]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:643)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:483)
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:413)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1106)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:886)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:819)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.cloud.LIROnShardRestartTest.testAllReplicasInLIR(LIROnShardRestartTest.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[GitHub] lucene-solr pull request #370: SOLR-12312: Change buf to not always use up 1...

[millerjeff0]

GitHub user millerjeff0 opened a pull request:

https://github.com/apache/lucene-solr/pull/370

SOLR-12312: Change buf to not always use up 1 MB

A lot of replicated files aren't 1 MB in size, this causes a lot of heap 
space waste when we create this for every file, instead create buffer based on 
the file size with a max of 1 MB

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/millerjeff0/lucene-solr SOLR-12312

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/lucene-solr/pull/370.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #370


commit 3499667b1b88bef207ab5825963e54db62d5eb2c
Author: Jeff 
Date:   2018-05-04T19:00:31Z

SOLR-12312: Change buf to not always use up 1 MB




---

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-MacOSX (64bit/jdk1.8.0) - Build # 615 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-MacOSX/615/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.testRestoreState

Error Message:
Did not expect the processor to fire on first run! event={   
"id":"26f452058311T5y07hug7nnzqp0bs17x6hoi9i",   "source":"node_added_trigger", 
  "eventTime":42830789968657,   "eventType":"NODEADDED",   "properties":{ 
"eventTimes":[   42830789968657,   42830789971036,   
42830789972528,   42830789973980], "nodeNames":[   
"127.0.0.1:56752_solr",   "127.0.0.1:56768_solr",   
"127.0.0.1:56791_solr",   "127.0.0.1:56763_solr"]}}

Stack Trace:
java.lang.AssertionError: Did not expect the processor to fire on first run! 
event={
  "id":"26f452058311T5y07hug7nnzqp0bs17x6hoi9i",
  "source":"node_added_trigger",
  "eventTime":42830789968657,
  "eventType":"NODEADDED",
  "properties":{
"eventTimes":[
  42830789968657,
  42830789971036,
  42830789972528,
  42830789973980],
"nodeNames":[
  "127.0.0.1:56752_solr",
  "127.0.0.1:56768_solr",
  "127.0.0.1:56791_solr",
  "127.0.0.1:56763_solr"]}}
at 
__randomizedtesting.SeedInfo.seed([6DDD4B60E1F74812:A373EFF319CE3004]:0)
at org.junit.Assert.fail(Assert.java:93)
at 
org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.lambda$new$0(NodeAddedTriggerTest.java:49)
at 
org.apache.solr.cloud.autoscaling.NodeAddedTrigger.run(NodeAddedTrigger.java:161)
at 
org.apache.solr.cloud.autoscaling.NodeAddedTriggerTest.testRestoreState(NodeAddedTriggerTest.java:257)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (SOLR-12249) Grouping on a solr.TextField works in stand-alone but not in SolrCloud

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12249?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464257#comment-16464257
 ] 

Varun Thacker commented on SOLR-12249:
--

{quote} propose that we explicitly dis-allow this in both stand-alone and Cloud 
mode, especially now that there's the SortableTextField.
{quote}
+1

> Grouping on a solr.TextField works in stand-alone but not in SolrCloud
> --
>
> Key: SOLR-12249
> URL: https://issues.apache.org/jira/browse/SOLR-12249
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.6.2, master (8.0)
>Reporter: Erick Erickson
>Priority: Minor
>
> I didn't test this with master. Under the covers in stand-alone mode, the 
> "min" function is silently  substituted for the grouping, but that's not true 
> in SorlCloud mode. I broke this JIRA out separately to discuss whether it 
> _ever_ makes sense to group by a tokenized text field.
> Grouping by the min value in a field is at least consistent, but on a text 
> field I don't think it makes sense.
> I propose that we explicitly dis-allow this in both stand-alone and Cloud 
> mode, especially now that there's the SortableTextField.
> Comments?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10169) PeerSync will hit an NPE on no response errors when looking for fingerprint.

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464246#comment-16464246
 ] 

Varun Thacker commented on SOLR-10169:
--

Hi [~erickerickson] should we backport this to Solr 7.3.1 ?

> PeerSync will hit an NPE on no response errors when looking for fingerprint.
> 
>
> Key: SOLR-10169
> URL: https://issues.apache.org/jira/browse/SOLR-10169
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Affects Versions: 6.6, 7.0
>Reporter: Mark Miller
>Assignee: Erick Erickson
>Priority: Major
> Fix For: 7.4
>
> Attachments: SOLR-10169.patch, SOLR-10169.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11884) find/fix inefficiencies in our use of logging

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464233#comment-16464233
 ] 

Varun Thacker commented on SOLR-11884:
--

> NOTE: SOLR-7887 fixed the %C issues, or at least when I checked on 27-Mar 
> there were no instances of %C in any of the log4j configuration files.

 

Confirmed. The performance implications of %C were documented 
[https://logging.apache.org/log4j/2.x/manual/layouts.html#LocationInformation] 
and up until Solr 7.3 the example log4j.properties had %C ( 
[https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.3.0/solr/example/resources/log4j.properties#L18]
 ) but we no longer have any %C starting Solr 7.4 

Can we close out this issue?

> find/fix inefficiencies in our use of logging
> -
>
> Key: SOLR-11884
> URL: https://issues.apache.org/jira/browse/SOLR-11884
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: logging
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
>
> We've been looking at Solr using Flight Recorder and ran across some 
> interesting things I'd like to discuss. Let's discuss general logging 
> approaches here, then perhaps break out sub-JIRAs when we reach any kind of 
> agreement.
> 1> Every log message generates a new Throwable, presumably to get things like 
> line number, file, class name and the like. On a 2 minute run blasting 
> updates this meant 150,000 (yes, 150K) instances of "new Throwable()".
>  
> See the section "Asynchronous Logging with Caller Location Information" at:
> [https://logging.apache.org/log4j/2.x/performance.html]
> I'm not totally sure changing the layout pattern will fix this in log4j 1.x, 
> but apparently certainly should in log4j 2.
>  
> The cost of course would be that lots of our log messages would lack some of 
> the information. Exceptions would still contain all the file/class/line 
> information of course.
>  
> Proposal:
> Change the layout pattern to, by default, _NOT_  include information that 
> requires a Throwable to be created. Also include a pattern that could be 
> un-commented to get this information back for troubleshooting.
>  
> 
>  
> We generate strings when we don't need them. Any construct like
> log.info("whatever " + method_that_builds_a_string + " : " + some_variable);
> generates the string (some of which are quite expensive) and then throws it 
> away if the log level is at, say, WARN. The above link also shows that 
> parameterizing this doesn't suffer this problem, so anything like the above 
> should be re-written as:
> log.info("whatever {} : {} ", method_that_builds_a_string, some_variable);
>  
> The alternative is to do something like but let's make use of the built-in 
> capabilities instead.
> if (log.level >= INFO) {
>    log.info("whatever " + method_that_builds_a_string + " : " + 
> some_variable);
> }
> etc.
> This would be a pretty huge thing to fix all-at-once so I suppose we'll have 
> to approach it incrementally. It's also something that, if we get them all 
> out of the code should be added to precommit failures. In the meantime, if 
> anyone who has the precommit chops could create a target that checked for 
> this it'd be a great help in tracking all of them down, then could be 
> incorporated in the regular precommit checks if/when they're all removed.
> Proposal:
> Use JFR or whatever to identify the egregious violations of this kind of 
> thing (I have a couple I've found) and change them to parameterized form (and 
> prove it works). Then see what we can do to move forward with removing them 
> all through the code base.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12312) Replication's IndexFetcher buf size can be initialized smarter to not waste RAM/GC

[David Smiley (JIRA)]

David Smiley created SOLR-12312:
---

 Summary: Replication's IndexFetcher buf size can be initialized 
smarter to not waste RAM/GC
 Key: SOLR-12312
 URL: https://issues.apache.org/jira/browse/SOLR-12312
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
  Components: replication (java)
Reporter: David Smiley


IndexFetcher's constructor knows the size of the file it's going to transfer.  
As-such, it ought to initialize the "buf" field to no larger than this size.  
This has been shown to waste Java heap/GC in  an environment with lots of cores 
of small indexes and thus small files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-12311) Suggester is not getting built on all replicas when "suggest.build=true" is issued

[Kannan Ranganathan (JIRA)]

Kannan Ranganathan created SOLR-12311:
-

 Summary: Suggester is not getting built on all replicas when 
"suggest.build=true" is issued
 Key: SOLR-12311
 URL: https://issues.apache.org/jira/browse/SOLR-12311
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
  Components: Suggester
Affects Versions: 7.3
Reporter: Kannan Ranganathan


The suggester dictionary is not getting built in all the replicas when a 
"suggest.build=true" is issued. It is getting built only on the replica that 
the first "suggest.build=true" query hits. Further queries that use the suggest 
component get only partial suggest results when the replicas where the 
dictionary is not built are hit.

This can be reproduced with the sample "techproducts" collection,
 # Create the "techproducts" collection with 2 shards and 2 replicas.
 # The default suggest component "mySuggester" has "buildOnStartup"=false
 # Send in this query to build the suggester and query it, 
"http://localhost:8983/solr/techproducts/suggest?suggest.build=true=mySuggester=elec;
 . You will see 4 suggestions.
 # Hit this query, without the "suggest.build=true" parameter multiple times 
and sometimes you will see 4 suggestions and in other times only 2 suggestions 
"http://localhost:8983/solr/techproducts/suggest?suggest.dictionary=mySuggester=elec;
 # When the above query in Step 4 is sent with "distrib=false" to each of the 
replicas, we can see that some replicas does not return any results.
 # When the logs are analyzed, we can see that the first time we send a query 
with "suggest.build=true", the suggest dictionary is built only on the replica 
that the distributed query hits and not the other ones.

Expected behaviour:

With one "suggest.build=true" query, the dictionary should be built on all 
replicas, so that further queries can get all the suggestions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11453) Create separate logger for slow requests

[Varun Thacker (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464166#comment-16464166
 ] 

Varun Thacker commented on SOLR-11453:
--

Hi Shawn,

Is this still on your radar. Now that we have successfully upgraded to Log4j2 
this wold be a useful feature to have enabled by default

> Create separate logger for slow requests
> 
>
> Key: SOLR-11453
> URL: https://issues.apache.org/jira/browse/SOLR-11453
> Project: Solr
>  Issue Type: Improvement
>  Components: logging
>Affects Versions: 7.0.1
>Reporter: Shawn Heisey
>Assignee: Shawn Heisey
>Priority: Minor
> Attachments: SOLR-11453.patch, SOLR-11453.patch, SOLR-11453.patch
>
>
> There is some desire on the mailing list to create a separate logfile for 
> slow queries.  Currently it is not possible to do this cleanly, because the 
> WARN level used by slow query logging within the SolrCore class is also used 
> for other events that SolrCore can log.  Those messages would be out of place 
> in a slow query log.  They should typically stay in main solr logfile.
> I propose creating a custom logger for slow queries, similar to what has been 
> set up for request logging.  In the SolrCore class, which is 
> org.apache.solr.core.SolrCore, there is a special logger at 
> org.apache.solr.core.SolrCore.Request.  This is not a real class, just a 
> logger which makes it possible to handle those log messages differently than 
> the rest of Solr's logging.  I propose setting up another custom logger 
> within SolrCore which could be org.apache.solr.core.SolrCore.SlowRequest.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12055) Enable asynch logging by default

[Varun Thacker (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Varun Thacker updated SOLR-12055:
-
Affects Version/s: (was: 7.2)

> Enable asynch logging by default
> 
>
> Key: SOLR-12055
> URL: https://issues.apache.org/jira/browse/SOLR-12055
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: logging
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Minor
>
> When SOLR-7887 is done, switching to asynch logging will be a simple change 
> to the config files for log4j2. This will reduce contention and increase 
> throughput generally and logging in particular.
> There's a discussion of the pros/cons here: 
> https://logging.apache.org/log4j/2.0/manual/async.html
> An alternative is to put a note in the Ref Guide about how to enable async 
> logging.
> I guess even if we enable async by default the ref guide still needs a note 
> about how to _disable_ it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk1.8.0) - Build # 4605 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/4605/
Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseG1GC

6 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger

Error Message:
waitFor not elapsed but produced an event

Stack Trace:
java.lang.AssertionError: waitFor not elapsed but produced an event
at 
__randomizedtesting.SeedInfo.seed([799AB1F4F1051E51:1A51877668CA6D7C]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.assertTrue(Assert.java:43)
at org.junit.Assert.assertNull(Assert.java:551)
at 
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testTrigger(IndexSizeTriggerTest.java:180)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.lang.Thread.run(Thread.java:748)


FAILED:  
org.apache.solr.cloud.autoscaling.IndexSizeTriggerTest.testMergeIntegration

Error Message:
did not finish processing in time

Stack Trace:
java.lang.AssertionError: did not finish processing in 

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-9.0.4) - Build # 21956 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/21956/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseParallelGC

1 tests failed.
FAILED:  org.apache.solr.cloud.autoscaling.SearchRateTriggerTest.testTrigger

Error Message:
expected:<1> but was:<2>

Stack Trace:
java.lang.AssertionError: expected:<1> but was:<2>
at 
__randomizedtesting.SeedInfo.seed([1CE770D1FE5D71F5:7F2C4653679202D8]:0)
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at 
org.apache.solr.cloud.autoscaling.SearchRateTriggerTest.testTrigger(SearchRateTriggerTest.java:133)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)




Build Log:
[...truncated 14339 lines...]
   [junit4] Suite: org.apache.solr.cloud.autoscaling.SearchRateTriggerTest
   [junit4]   2> 

[jira] [Commented] (LUCENE-7976) Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of very large segments

[Erick Erickson (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464131#comment-16464131
 ] 

Erick Erickson commented on LUCENE-7976:


Simon:

Thanks for taking the time, and yeah, the complexity bothers me too.

I'm reluctant to take the comments out for 8263 as their lifetime is expected 
to be a few days after this is checked in and I don't want to have to remember 
how I got it to work, but we can if you strongly object.

About tests. There are actually a number of tests for TMP functionality 
already, in particular merging down to N segments and forcemerging, what 
particular parts of the code do you think should be exercised more?

As for naming/style changes, sure. I always feel like an interloper in the 
Lucene code;  I'm perfectly willing to try to keep it consistent with the 
folks' expectations who, you know live in the Lucene codebase ;)

As for your other suggestions, I don't know. I've rewritten/refactored/whatever 
this about three times already and every time I do it gets complicated again.

bq. I wonder if we can generalize the algorithm here into a single method 
because in the end they all do the same thing.

doFindMerges is called from all three operations so I'm a bit puzzled about 
what this would look like. Each of the three operations has different initial 
conditions, eligible segments, max segment size allowed and the like. Once 
those are established, they all go through the same scoring/selection code. 
Figuring out the initial conditions is "interesting"

There are several areas here that are particularly gnarly and any way to 
un-gnarl them would be great.

> the variable number of segments in findForcedMerges. I'd love for there to be 
> exactly two choices; merge down to one or respect max segment size. 
> Controlling the number of segments would then be more along the lines of 
> setting maxMergedSegmentsMB in TMP.

> the multi-pass nature of findForcedMerges because it respects the 
> maxMergeAtOnceExplicit.

> Somewhat different decisions need to be made in doFindMerges depending on 
> what type of merge it is. I'm not a huge fan of passing that enum in. One of 
> the iterations tried to pass information into an uber-function but that lead 
> to having to pass around segmentsToMerge from findForcedMerges, which wasn't 
> present in the other two, so passing null in from them was also ugly.

> I also ran into a couple of issues with findMerges needing to _not_ merge 
> segments if there weren't enough in the tier, which is exactly the opposite 
> of findForcedMerges, which itself has conditions around whether it should 
> merge a segment with no deletions or not if exceeds maxMergedSegmentMB which 
> itself is a variable condition based on whether maxSegments has been 
> specified.

Let me know if you're going to take a whack at it, even a skeleton of a 
different approach would help me get on the same page.

Meanwhile I can incorporate your other comments, they'll be useful no matter 
what.

> Make TieredMergePolicy respect maxSegmentSizeMB and allow singleton merges of 
> very large segments
> -
>
> Key: LUCENE-7976
> URL: https://issues.apache.org/jira/browse/LUCENE-7976
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Erick Erickson
>Assignee: Erick Erickson
>Priority: Major
> Attachments: LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, 
> LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch, LUCENE-7976.patch
>
>
> We're seeing situations "in the wild" where there are very large indexes (on 
> disk) handled quite easily in a single Lucene index. This is particularly 
> true as features like docValues move data into MMapDirectory space. The 
> current TMP algorithm allows on the order of 50% deleted documents as per a 
> dev list conversation with Mike McCandless (and his blog here:  
> https://www.elastic.co/blog/lucenes-handling-of-deleted-documents).
> Especially in the current era of very large indexes in aggregate, (think many 
> TB) solutions like "you need to distribute your collection over more shards" 
> become very costly. Additionally, the tempting "optimize" button exacerbates 
> the issue since once you form, say, a 100G segment (by 
> optimizing/forceMerging) it is not eligible for merging until 97.5G of the 
> docs in it are deleted (current default 5G max segment size).
> The proposal here would be to add a new parameter to TMP, something like 
>  (no, that's not serious name, suggestions 
> welcome) which would default to 100 (or the same behavior we have now).
> So if I set this parameter to, say, 20%, and the max segment size stays at 
> 5G, the following would happen when segments were selected for merging:
> > any segment 

[jira] [Updated] (SOLR-11277) Add auto hard commit setting based on tlog size

[Anshum Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anshum Gupta updated SOLR-11277:

Fix Version/s: master (8.0)

> Add auto hard commit setting based on tlog size
> ---
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Rupa Shankar
>Assignee: Anshum Gupta
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> max_size_auto_commit.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can 
> be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. 
> We’ve had some occurrences of really huge tlogs, resulting in serious issues, 
> so in an attempt to avoid this, it would be great to have a “maxSize” setting 
> based on the tlog size on disk. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-11277) Add auto hard commit setting based on tlog size

[Anshum Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anshum Gupta updated SOLR-11277:

Fix Version/s: 7.4

> Add auto hard commit setting based on tlog size
> ---
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Rupa Shankar
>Assignee: Anshum Gupta
>Priority: Major
> Fix For: 7.4
>
> Attachments: SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> max_size_auto_commit.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can 
> be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. 
> We’ve had some occurrences of really huge tlogs, resulting in serious issues, 
> so in an attempt to avoid this, it would be great to have a “maxSize” setting 
> based on the tlog size on disk. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-11277) Add auto hard commit setting based on tlog size

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464051#comment-16464051
 ] 

ASF subversion and git services commented on SOLR-11277:


Commit 6ec1198a5144e73b47bc88cd79f534878cbcbef4 in lucene-solr's branch 
refs/heads/branch_7x from [~anshumg]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=6ec1198 ]

SOLR-11277: Add auto hard commit setting based on tlog size (this closes #358)


> Add auto hard commit setting based on tlog size
> ---
>
> Key: SOLR-11277
> URL: https://issues.apache.org/jira/browse/SOLR-11277
> Project: Solr
>  Issue Type: New Feature
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Rupa Shankar
>Assignee: Anshum Gupta
>Priority: Major
> Attachments: SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, SOLR-11277.patch, 
> max_size_auto_commit.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When indexing documents of variable sizes and at variable schedules, it can 
> be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. 
> We’ve had some occurrences of really huge tlogs, resulting in serious issues, 
> so in an attempt to avoid this, it would be great to have a “maxSize” setting 
> based on the tlog size on disk. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12258) V2 API should "retry" for unresolved collections/aliases (like V1 does)

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464033#comment-16464033
 ] 

David Smiley commented on SOLR-12258:
-

The attached patch:
 * V2HttpCall: moved the alias resolution into getDocCollection and renamed to 
resolveDocCollection to handle both alias & collection resolution.  Added retry 
logic, both for aliases and for collections (via  
ZkStateReader.forceUpdateCollection(collection)
 * HttpSolrCall: after we call aliases.update(), added a call to 
{{cores.getZkController().zkStateReader.forceUpdateCollection(collectionName);}}
  And I removed the comment about admin UI elements triggering this code path 
because this is no longer true – I checked by using Solr with a breakpoint both 
cloud and standard, and by researching the original issue.
 * ZkStateReader.forceUpdateCollection: added TODO comment that we ought to 
call ZooKeeper.sync().

Tests are running now.

Any comments [~gus_heck], [~noble.paul] or [~shalinmangar]?  If not I'll commit 
this maybe Monday.  I'm looking forward to not seeing spurious CI failures for 
TimeRoutedAliasUpdateProcessorTest that will be solved by this.

> V2 API should "retry" for unresolved collections/aliases (like V1 does)
> ---
>
> Key: SOLR-12258
> URL: https://issues.apache.org/jira/browse/SOLR-12258
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud, v2 API
>Reporter: David Smiley
>Priority: Major
> Attachments: SOLR-12258.patch, SOLR-12258.patch
>
>
> When using V1, if the request refers to a possible collection/alias that 
> fails to resolve, HttpSolrCall will invoke AliasesManager.update() then retry 
> the request as if anew (in collaboration with SolrDispatchFilter).  If it 
> fails to resolve again we stop there and return an error; it doesn't go on 
> forever.
> V2 (V2HttpCall specifically) doesn't have this retry mechanism.  It'll return 
> "no such collection or alias".
> The retry will not only work for an alias but the retrying is a delay that 
> will at least help the odds of a newly made collection from being known to 
> this Solr node.  It'd be nice if this was more explicit – i.e. if there was a 
> mechanism similar to AliasesManager.update() but for a collection.  I'm not 
> sure how to do that?
> BTW I discovered this while debugging a Jenkins failure of 
> TimeRoutedAliasUpdateProcessorTest.test where it early on simply goes to 
> issue a V2 based request to change the configuration of a collection that was 
> created immediately before it.  It's pretty mysterious.  I am aware of 
> SolrCloudTestCase.waitForState which is maybe something that needs to be 
> called?  But if that were true then *every* SolrCloud test would need to use 
> it; it just seems wrong to me that we ought to use this method commonly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12258) V2 API should "retry" for unresolved collections/aliases (like V1 does)

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated SOLR-12258:

Attachment: SOLR-12258.patch

> V2 API should "retry" for unresolved collections/aliases (like V1 does)
> ---
>
> Key: SOLR-12258
> URL: https://issues.apache.org/jira/browse/SOLR-12258
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud, v2 API
>Reporter: David Smiley
>Priority: Major
> Attachments: SOLR-12258.patch, SOLR-12258.patch
>
>
> When using V1, if the request refers to a possible collection/alias that 
> fails to resolve, HttpSolrCall will invoke AliasesManager.update() then retry 
> the request as if anew (in collaboration with SolrDispatchFilter).  If it 
> fails to resolve again we stop there and return an error; it doesn't go on 
> forever.
> V2 (V2HttpCall specifically) doesn't have this retry mechanism.  It'll return 
> "no such collection or alias".
> The retry will not only work for an alias but the retrying is a delay that 
> will at least help the odds of a newly made collection from being known to 
> this Solr node.  It'd be nice if this was more explicit – i.e. if there was a 
> mechanism similar to AliasesManager.update() but for a collection.  I'm not 
> sure how to do that?
> BTW I discovered this while debugging a Jenkins failure of 
> TimeRoutedAliasUpdateProcessorTest.test where it early on simply goes to 
> issue a V2 based request to change the configuration of a collection that was 
> created immediately before it.  It's pretty mysterious.  I am aware of 
> SolrCloudTestCase.waitForState which is maybe something that needs to be 
> called?  But if that were true then *every* SolrCloud test would need to use 
> it; it just seems wrong to me that we ought to use this method commonly.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8296) PendingDeletes shouldn't write to live docs that it shared

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand updated LUCENE-8296:
-
Attachment: LUCENE-8296.patch

> PendingDeletes shouldn't write to live docs that it shared
> --
>
> Key: LUCENE-8296
> URL: https://issues.apache.org/jira/browse/LUCENE-8296
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-8296.patch
>
>
> PendingDeletes has a markAsShared mechanism that allow to make sure that the 
> latest livedocs are not going to receive more updates. But it is not always 
> used, and I was able to verify that in some cases we end up with readers 
> whose live docs disagree with the number of deletes. Even though this might 
> not be causing bugs, it feels dangerous to me so I think we should consider 
> always marking live docs as shared in #getLiveDocs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8296) PendingDeletes shouldn't write to live docs that it shared

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464009#comment-16464009
 ] 

Adrien Grand commented on LUCENE-8296:
--

Here is a patch.

> PendingDeletes shouldn't write to live docs that it shared
> --
>
> Key: LUCENE-8296
> URL: https://issues.apache.org/jira/browse/LUCENE-8296
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-8296.patch
>
>
> PendingDeletes has a markAsShared mechanism that allow to make sure that the 
> latest livedocs are not going to receive more updates. But it is not always 
> used, and I was able to verify that in some cases we end up with readers 
> whose live docs disagree with the number of deletes. Even though this might 
> not be causing bugs, it feels dangerous to me so I think we should consider 
> always marking live docs as shared in #getLiveDocs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8296) PendingDeletes shouldn't write to live docs that it shared

[Adrien Grand (JIRA)]

Adrien Grand created LUCENE-8296:


 Summary: PendingDeletes shouldn't write to live docs that it shared
 Key: LUCENE-8296
 URL: https://issues.apache.org/jira/browse/LUCENE-8296
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Adrien Grand


PendingDeletes has a markAsShared mechanism that allow to make sure that the 
latest livedocs are not going to receive more updates. But it is not always 
used, and I was able to verify that in some cases we end up with readers whose 
live docs disagree with the number of deletes. Even though this might not be 
causing bugs, it feels dangerous to me so I think we should consider always 
marking live docs as shared in #getLiveDocs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8998) JSON Facet API child roll-ups

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463988#comment-16463988
 ] 

Cassandra Targett commented on SOLR-8998:
-

[~mkhludnev], sorry for the delay taking a look at this. Just to make sure I 
understand what you asked me to look at...Some documentation was done with the 
code changes last week, but then there are some additional doc changes that are 
proposed in the {{SOLR-8998-doc.patch}}, is that correct?

I haven't reviewed in much depth yet, but one thing that jumped out at me right 
away is if you look at the BlockJoin Faceting page as it's built 
(https://builds.apache.org/view/L/view/Lucene/job/Solr-reference-guide-master/javadoc/blockjoin-faceting.html)
 there are two warnings near the top of the page. The one that was added with 
this issue says the functionality is deprecated, but the next one that's been 
there for a while says it's experimental. Those statements on the status 
conflict, I think - "experimental" would indicate to me that future development 
is likely, but "deprecated" says no more development should be expected. I 
think we could merge those statements with something like:

{noformat}
This functionality is considered deprecated. Users are encouraged to use 
uniqueBlock(_root_) aggregation under terms facet in JSON Facet API.

If this component is used, it must be explicitly enabled for a request handler 
in solrconfig.xml, in the same way as any other search component.
{noformat}

Does that seem reasonable and an accurate reflection of the status of 
{{BlockJoinFacetComponent}}?

> JSON Facet API child roll-ups
> -
>
> Key: SOLR-8998
> URL: https://issues.apache.org/jira/browse/SOLR-8998
> Project: Solr
>  Issue Type: New Feature
>  Components: Facet Module
>Reporter: Yonik Seeley
>Assignee: Mikhail Khludnev
>Priority: Major
> Fix For: 7.4, master (8.0)
>
> Attachments: SOLR-8998-api-doc.patch, SOLR-8998-doc.patch, 
> SOLR-8998.patch, SOLR-8998.patch, SOLR-8998.patch, SOLR_8998.patch, 
> SOLR_8998.patch, SOLR_8998.patch
>
>
> The JSON Facet API currently has the ability to map between parents and 
> children ( see http://yonik.com/solr-nested-objects/ )
> This issue is about adding a true rollup ability where parents would take on 
> derived values from their children.  The most important part (and the most 
> difficult part) will be the external API.
> [~mkhludnev] says
> {quote}
> The bottom line is to introduce {{uniqueBlock(\_root_)}} aggregation, which 
> is supposed to be faster than {{unique(\_root_)}} and purposed for blocked 
> index only. For now it it supports singlevalue string fields, docValues 
> usually make sense.   
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Windows (32bit/jdk1.8.0_144) - Build # 576 - Still Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Windows/576/
Java: 32bit/jdk1.8.0_144 -client -XX:+UseSerialGC

15 tests failed.
FAILED:  org.apache.solr.handler.component.SpellCheckComponentTest.testCollate

Error Message:
Directory 
(SimpleFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.component.SpellCheckComponentTest_168EC6E8309F719C-001\init-core-data-001\spellchecker1
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@17ad893) still has 
pending deleted files; cannot initialize IndexWriter

Stack Trace:
java.lang.IllegalArgumentException: Directory 
(SimpleFSDirectory@C:\Users\jenkins\workspace\Lucene-Solr-7.x-Windows\solr\build\solr-core\test\J1\temp\solr.handler.component.SpellCheckComponentTest_168EC6E8309F719C-001\init-core-data-001\spellchecker1
 lockFactory=org.apache.lucene.store.NativeFSLockFactory@17ad893) still has 
pending deleted files; cannot initialize IndexWriter
at 
__randomizedtesting.SeedInfo.seed([168EC6E8309F719C:DCAF9FDC9A674CF7]:0)
at org.apache.lucene.index.IndexWriter.(IndexWriter.java:699)
at 
org.apache.lucene.search.spell.SpellChecker.clearIndex(SpellChecker.java:455)
at 
org.apache.solr.spelling.IndexBasedSpellChecker.build(IndexBasedSpellChecker.java:87)
at 
org.apache.solr.handler.component.SpellCheckComponent.prepare(SpellCheckComponent.java:128)
at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:269)
at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:2510)
at org.apache.solr.util.TestHarness.query(TestHarness.java:337)
at org.apache.solr.util.TestHarness.query(TestHarness.java:319)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:982)
at org.apache.solr.SolrTestCaseJ4.assertJQ(SolrTestCaseJ4.java:951)
at 
org.apache.solr.handler.component.SpellCheckComponentTest.testCollate(SpellCheckComponentTest.java:171)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1737)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:934)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:970)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:984)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:943)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:829)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:879)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:890)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 

[jira] [Commented] (LUCENE-7960) NGram filters -- add option to keep short terms

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-7960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463930#comment-16463930
 ] 

Shawn Heisey commented on LUCENE-7960:
--

On first blush, an enum seems even more of a mess than one or two extra boolean 
parameters.  I will let [~rcmuir] and others with more experience in these 
matters make the call on that.  How would the factory (and by extension, text 
configuration files) handle it?

If two boolean parameters is going to meet with resistance, I can support 
preserveOriginal.  I think users will want long and short handled separately, 
but one flag would get the job done.


> NGram filters -- add option to keep short terms
> ---
>
> Key: LUCENE-7960
> URL: https://issues.apache.org/jira/browse/LUCENE-7960
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Shawn Heisey
>Priority: Major
> Attachments: LUCENE-7960.patch, LUCENE-7960.patch
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> When ngram or edgengram filters are used, any terms that are shorter than the 
> minGramSize are completely removed from the token stream.
> This is probably 100% what was intended, but I've seen it cause a lot of 
> problems for users.  I am not suggesting that the default behavior be 
> changed.  That would be far too disruptive to the existing user base.
> I do think there should be a new boolean option, with a name like 
> keepShortTerms, that defaults to false, to allow the short terms to be 
> preserved.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12303) Subquery Doc transform doesn't inherit path from original request

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463855#comment-16463855
 ] 

Munendra S N commented on SOLR-12303:
-

 [^SOLR-12303.patch] 
Test to verify if parent path is inherited or not.  Created a separate config 
without */select* to test this

> Subquery Doc transform doesn't inherit path from original request
> -
>
> Key: SOLR-12303
> URL: https://issues.apache.org/jira/browse/SOLR-12303
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Munendra S N
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-12303.patch, SOLR-12303.patch
>
>
> {code:java}
> localhost:8983/solr/k_test/search?sort=score desc,uniqueId 
> desc=AND=json={!parent which=parent_field:true score=max}({!edismax 
> v=$origQuery})=false=uniqueId=score=_children_:[subquery]=uniqueId=false=parent_field&_children_.fl=uniqueId&_children_.fl=score&_children_.rows=3=false&_children_.q={!edismax
>  qf=parentId v=$row.uniqueId}=1
> {code}
> For this request, even though the path is */search*, the subquery request 
> would be fired on handler */select*.
> Subquery request should inherit the parent request handler and there should 
> be an option to override this behavior. (option to override is already 
> available by specifying *qt*)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12303) Subquery Doc transform doesn't inherit path from original request

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N updated SOLR-12303:

Attachment: SOLR-12303.patch

> Subquery Doc transform doesn't inherit path from original request
> -
>
> Key: SOLR-12303
> URL: https://issues.apache.org/jira/browse/SOLR-12303
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Munendra S N
>Assignee: Mikhail Khludnev
>Priority: Major
> Attachments: SOLR-12303.patch, SOLR-12303.patch
>
>
> {code:java}
> localhost:8983/solr/k_test/search?sort=score desc,uniqueId 
> desc=AND=json={!parent which=parent_field:true score=max}({!edismax 
> v=$origQuery})=false=uniqueId=score=_children_:[subquery]=uniqueId=false=parent_field&_children_.fl=uniqueId&_children_.fl=score&_children_.rows=3=false&_children_.q={!edismax
>  qf=parentId v=$row.uniqueId}=1
> {code}
> For this request, even though the path is */search*, the subquery request 
> would be fired on handler */select*.
> Subquery request should inherit the parent request handler and there should 
> be an option to override this behavior. (option to override is already 
> available by specifying *qt*)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Query on searchAfter API usage in IndexSearcher

[manish gupta]

Hi Team,

I am new to Lucene and I am trying to use Lucene for text search in my
project to achieve better results in terms of query performance.

Initially I was facing lot of GC issues while using lucene as I was using
search API and passing all the documents count. As my data size is around 4
billion the number of documents created by Lucene were huge. Internally
search API uses TopScoreDocCollector which internally creates a
PriorityQueue of given documents count thus causing lot of GC.

*To avoid this problem I am trying to query using a pagination way wherein
I am query only 10 documents at a time and after that I am using
seacrhAfter API to query further passing the lastScoreDoc from previous
result. This has resolved the GC problem but the query time has increased
by a huge margin from 3 sec to 600 sec.*

*When I debugged I found that even though I use the searchAfter API, it is
not avoiding the IO and every time it is reading the data from disk again.
It is only skipping the results filled in previous search. Is my
understanding correct?. If yes please let me know if there is a better way
to query the results in incremental order so as to avoid GC and with
minimal impact on query performance.*

Regards
Manish Gupta

[jira] [Commented] (LUCENE-8295) Remove ReadersAndUpdates.liveDocsSharedPending

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463835#comment-16463835
 ] 

ASF subversion and git services commented on LUCENE-8295:
-

Commit 27992e3386f7ba9521329ec8a1957103d73da2e4 in lucene-solr's branch 
refs/heads/branch_7x from [~jpountz]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=27992e3 ]

LUCENE-8295: Remove useless liveDocsSharedPending flag.


> Remove ReadersAndUpdates.liveDocsSharedPending
> --
>
> Key: LUCENE-8295
> URL: https://issues.apache.org/jira/browse/LUCENE-8295
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8295.patch
>
>
> I have been trying to undersdand PendingDeletes and ReadersAndUpdates, and it 
> looks to me that the liveDocsSharedPending flag doesn't buy anything. I ran 
> tests 10 times after removing it and got no failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8295) Remove ReadersAndUpdates.liveDocsSharedPending

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463836#comment-16463836
 ] 

ASF subversion and git services commented on LUCENE-8295:
-

Commit ad0ad2ec891b7c122ddf5b0385a9e4b984e78b38 in lucene-solr's branch 
refs/heads/master from [~jpountz]
[ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=ad0ad2e ]

LUCENE-8295: Remove useless liveDocsSharedPending flag.


> Remove ReadersAndUpdates.liveDocsSharedPending
> --
>
> Key: LUCENE-8295
> URL: https://issues.apache.org/jira/browse/LUCENE-8295
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8295.patch
>
>
> I have been trying to undersdand PendingDeletes and ReadersAndUpdates, and it 
> looks to me that the liveDocsSharedPending flag doesn't buy anything. I ran 
> tests 10 times after removing it and got no failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8295) Remove ReadersAndUpdates.liveDocsSharedPending

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8295.
--
   Resolution: Fixed
Fix Version/s: master (8.0)
   7.4

> Remove ReadersAndUpdates.liveDocsSharedPending
> --
>
> Key: LUCENE-8295
> URL: https://issues.apache.org/jira/browse/LUCENE-8295
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Fix For: 7.4, master (8.0)
>
> Attachments: LUCENE-8295.patch
>
>
> I have been trying to undersdand PendingDeletes and ReadersAndUpdates, and it 
> looks to me that the liveDocsSharedPending flag doesn't buy anything. I ran 
> tests 10 times after removing it and got no failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-12238) Synonym Query Style Boost By Payload

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463827#comment-16463827
 ] 

Alessandro Benedetti edited comment on SOLR-12238 at 5/4/18 12:52 PM:
--

i just attached the latest version of the patch ( apparently there may be some 
problems with the patch autogenerated from Github).
 I wasn't able to reproduce the Junit tests failures Jenkins reported ( and 
they appeared to me to be quite unrelated)


was (Author: alessandro.benedetti):
i just attached the latest version of the patch ( apparently there may be some 
problems with the patch autogenerated from Github).
I was able to reproduce the Junit tests failures Jenkins reported ( and they 
appeared to me to be quite unrelated)

> Synonym Query Style Boost By Payload
> 
>
> Key: SOLR-12238
> URL: https://issues.apache.org/jira/browse/SOLR-12238
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.2
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12238.patch, SOLR-12238.patch, SOLR-12238.patch, 
> SOLR-12238.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This improvement is built on top of the Synonym Query Style feature and 
> brings the possibility of boosting synonym queries using the payload 
> associated.
> It introduces two new modalities for the Synonym Query Style :
> PICK_BEST_BOOST_BY_PAYLOAD -> build a Disjunction query with the clauses 
> boosted by payload
> AS_DISTINCT_TERMS_BOOST_BY_PAYLOAD -> build a Boolean query with the clauses 
> boosted by payload
> This new synonym query styles will assume payloads are available so they must 
> be used in conjunction with a token filter able to produce payloads.
> An synonym.txt example could be :
> # Synonyms used by Payload Boost
> tiger => tiger|1.0, Big_Cat|0.8, Shere_Khan|0.9
> leopard => leopard, Big_Cat|0.8, Bagheera|0.9
> lion => lion|1.0, panthera leo|0.99, Simba|0.8
> snow_leopard => panthera uncia|0.99, snow leopard|1.0
> A simple token filter to populate the payloads from such synonym.txt is :
>  delimiter="|"/>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12243) Edismax missing phrase queries when phrases contain multiterm synonyms

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463831#comment-16463831
 ] 

Alessandro Benedetti commented on SOLR-12243:
-

Apparently there are some issues with auto generated github patches and the 
Jenkins job validating the patch.

In theory it should just work straightforward :
Github Pull Request -> patch extracted -> patch validated through Jenkins job

But the process is not smooth and I had to push various version of the same 
patch ( this is valid for others issues as well) .
Attaching the latest patch ( generated from git diff)

> Edismax missing phrase queries when phrases contain multiterm synonyms
> --
>
> Key: SOLR-12243
> URL: https://issues.apache.org/jira/browse/SOLR-12243
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.1
> Environment: RHEL, MacOS X
> Do not believe this is environment-specific.
>Reporter: Elizabeth Haubert
>Priority: Major
> Attachments: SOLR-12243.patch, SOLR-12243.patch, SOLR-12243.patch, 
> SOLR-12243.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> synonyms.txt:
> allergic, hypersensitive
> aspirin, acetylsalicylic acid
> dog, canine, canis familiris, k 9
> rat, rattus
> request handler:
> 
>  
> 
>  edismax
>   0.4
>  title^100
>  title~20^5000
>  title~11
>  title~22^1000
>  text
>  
>  3-1 6-3 930%
>  *:*
>  25
> 
>  
> Phrase queries (pf, pf2, pf3) containing "dog" or "aspirin"  against the 
> above list will not be generated.
> "allergic reaction dog" will generate pf2: "allergic reaction", but not 
> pf:"allergic reaction dog", pf2: "reaction dog", or pf3: "allergic reaction 
> dog"
> "aspirin dose in rats" will generate pf3: "dose ? rats" but not pf2: "aspirin 
> dose" or pf3:"aspirin dose ?"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12238) Synonym Query Style Boost By Payload

[Doug Turnbull (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463830#comment-16463830
 ] 

Doug Turnbull commented on SOLR-12238:
--

Just want to say I've been watching this feature and 

+1 - great feature!

Exactly the kind of thing I was hoping to see after much of [~ehatcher]'s great 
payload work :)

> Synonym Query Style Boost By Payload
> 
>
> Key: SOLR-12238
> URL: https://issues.apache.org/jira/browse/SOLR-12238
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.2
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12238.patch, SOLR-12238.patch, SOLR-12238.patch, 
> SOLR-12238.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This improvement is built on top of the Synonym Query Style feature and 
> brings the possibility of boosting synonym queries using the payload 
> associated.
> It introduces two new modalities for the Synonym Query Style :
> PICK_BEST_BOOST_BY_PAYLOAD -> build a Disjunction query with the clauses 
> boosted by payload
> AS_DISTINCT_TERMS_BOOST_BY_PAYLOAD -> build a Boolean query with the clauses 
> boosted by payload
> This new synonym query styles will assume payloads are available so they must 
> be used in conjunction with a token filter able to produce payloads.
> An synonym.txt example could be :
> # Synonyms used by Payload Boost
> tiger => tiger|1.0, Big_Cat|0.8, Shere_Khan|0.9
> leopard => leopard, Big_Cat|0.8, Bagheera|0.9
> lion => lion|1.0, panthera leo|0.99, Simba|0.8
> snow_leopard => panthera uncia|0.99, snow leopard|1.0
> A simple token filter to populate the payloads from such synonym.txt is :
>  delimiter="|"/>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12243) Edismax missing phrase queries when phrases contain multiterm synonyms

[Alessandro Benedetti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Benedetti updated SOLR-12243:

Attachment: SOLR-12243.patch

> Edismax missing phrase queries when phrases contain multiterm synonyms
> --
>
> Key: SOLR-12243
> URL: https://issues.apache.org/jira/browse/SOLR-12243
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.1
> Environment: RHEL, MacOS X
> Do not believe this is environment-specific.
>Reporter: Elizabeth Haubert
>Priority: Major
> Attachments: SOLR-12243.patch, SOLR-12243.patch, SOLR-12243.patch, 
> SOLR-12243.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> synonyms.txt:
> allergic, hypersensitive
> aspirin, acetylsalicylic acid
> dog, canine, canis familiris, k 9
> rat, rattus
> request handler:
> 
>  
> 
>  edismax
>   0.4
>  title^100
>  title~20^5000
>  title~11
>  title~22^1000
>  text
>  
>  3-1 6-3 930%
>  *:*
>  25
> 
>  
> Phrase queries (pf, pf2, pf3) containing "dog" or "aspirin"  against the 
> above list will not be generated.
> "allergic reaction dog" will generate pf2: "allergic reaction", but not 
> pf:"allergic reaction dog", pf2: "reaction dog", or pf3: "allergic reaction 
> dog"
> "aspirin dose in rats" will generate pf3: "dose ? rats" but not pf2: "aspirin 
> dose" or pf3:"aspirin dose ?"
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12238) Synonym Query Style Boost By Payload

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463827#comment-16463827
 ] 

Alessandro Benedetti commented on SOLR-12238:
-

i just attached the latest version of the patch ( apparently there may be some 
problems with the patch autogenerated from Github).
I was able to reproduce the Junit tests failures Jenkins reported ( and they 
appeared to me to be quite unrelated)

> Synonym Query Style Boost By Payload
> 
>
> Key: SOLR-12238
> URL: https://issues.apache.org/jira/browse/SOLR-12238
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.2
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12238.patch, SOLR-12238.patch, SOLR-12238.patch, 
> SOLR-12238.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This improvement is built on top of the Synonym Query Style feature and 
> brings the possibility of boosting synonym queries using the payload 
> associated.
> It introduces two new modalities for the Synonym Query Style :
> PICK_BEST_BOOST_BY_PAYLOAD -> build a Disjunction query with the clauses 
> boosted by payload
> AS_DISTINCT_TERMS_BOOST_BY_PAYLOAD -> build a Boolean query with the clauses 
> boosted by payload
> This new synonym query styles will assume payloads are available so they must 
> be used in conjunction with a token filter able to produce payloads.
> An synonym.txt example could be :
> # Synonyms used by Payload Boost
> tiger => tiger|1.0, Big_Cat|0.8, Shere_Khan|0.9
> leopard => leopard, Big_Cat|0.8, Bagheera|0.9
> lion => lion|1.0, panthera leo|0.99, Simba|0.8
> snow_leopard => panthera uncia|0.99, snow leopard|1.0
> A simple token filter to populate the payloads from such synonym.txt is :
>  delimiter="|"/>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12238) Synonym Query Style Boost By Payload

[Alessandro Benedetti (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alessandro Benedetti updated SOLR-12238:

Attachment: SOLR-12238.patch

> Synonym Query Style Boost By Payload
> 
>
> Key: SOLR-12238
> URL: https://issues.apache.org/jira/browse/SOLR-12238
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: query parsers
>Affects Versions: 7.2
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12238.patch, SOLR-12238.patch, SOLR-12238.patch, 
> SOLR-12238.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> This improvement is built on top of the Synonym Query Style feature and 
> brings the possibility of boosting synonym queries using the payload 
> associated.
> It introduces two new modalities for the Synonym Query Style :
> PICK_BEST_BOOST_BY_PAYLOAD -> build a Disjunction query with the clauses 
> boosted by payload
> AS_DISTINCT_TERMS_BOOST_BY_PAYLOAD -> build a Boolean query with the clauses 
> boosted by payload
> This new synonym query styles will assume payloads are available so they must 
> be used in conjunction with a token filter able to produce payloads.
> An synonym.txt example could be :
> # Synonyms used by Payload Boost
> tiger => tiger|1.0, Big_Cat|0.8, Shere_Khan|0.9
> leopard => leopard, Big_Cat|0.8, Bagheera|0.9
> lion => lion|1.0, panthera leo|0.99, Simba|0.8
> snow_leopard => panthera uncia|0.99, snow leopard|1.0
> A simple token filter to populate the payloads from such synonym.txt is :
>  delimiter="|"/>



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-12299) More Like This Params Refactor

[Alessandro Benedetti (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463794#comment-16463794
 ] 

Alessandro Benedetti edited comment on SOLR-12299 at 5/4/18 12:34 PM:
--

I am attaching the patch again.
Apparently the one generated automatically from the github request was 
partially corrupted.
Fixed now.


was (Author: alessandro.benedetti):
I am attaching the patch again, created from Github Pull Request.
I just double checked there and it does not show any conflict with the master, 
I will investigate what the jenkins job means.

> More Like This Params Refactor
> --
>
> Key: SOLR-12299
> URL: https://issues.apache.org/jira/browse/SOLR-12299
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: MoreLikeThis
>Reporter: Alessandro Benedetti
>Priority: Major
> Attachments: SOLR-12299.patch, SOLR-12299.patch, SOLR-12299.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> More Like This ca be refactored to improve the code readability, test 
> coverage and maintenance.
> Scope of this Jira issue is to start the More Like This refactor from the 
> More Like This Params.
> This Jira will not improve the current More Like This but just keep the same 
> functionality with a refactored code.
> Other Jira issues will follow improving the overall code readability, test 
> coverage and maintenance.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org