[GitHub] [lucene-solr] magibney commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

magibney commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302357020
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -449,9 +556,83 @@ private static boolean hasReplicaType(Object o, String 
preferred) {
 }
   }
 
+  private final ReplicaListTransformerFactory randomRltFactory = (String 
configSpec, SolrQueryRequest request,
+  ReplicaListTransformerFactory fallback) -> 
shufflingReplicaListTransformer;
+  private ReplicaListTransformerFactory stableRltFactory;
+  private ReplicaListTransformerFactory defaultRltFactory;
+
+  /**
+   * Private class responsible for applying pairwise sort based on inherent 
replica attributes,
+   * and subsequently reordering any equivalent replica sets according to 
behavior specified
+   * by the baseReplicaListTransformer.
+   */
+  private static final class TopLevelReplicaListTransformer implements 
ReplicaListTransformer {
+
+private final NodePreferenceRulesComparator replicaComp;
+private final ReplicaListTransformer baseReplicaListTransformer;
+
+public TopLevelReplicaListTransformer(NodePreferenceRulesComparator 
replicaComp, ReplicaListTransformer baseReplicaListTransformer) {
+  this.replicaComp = replicaComp;
+  this.baseReplicaListTransformer = baseReplicaListTransformer;
+}
+
+@Override
+public void transform(List choices) {
 
 Review comment:
   I had to think about this for a minute, when I wrote it, and again now :-) 
...
   
   I think it actually _would_ be different. The key is that the affinity code 
is always based on the same absolute order; it's only the _rotation_ that 
changes. And if we apply the rotation to the overall list before applying other 
preferences, the stability of the sort doesn't help us.
   
   Take for example a situation with replicas [A, B, C, D, E]. For the sake of 
argument, let's say that B and C will end up being tied for "most preferred", 
according to the inherent preference comparators. The "affinity" 
transformation, applied at the top level, would sort the replicas: [A, B, C, D, 
E], then rotate them according to some explicitly specified or hash-derived int 
param. The problem is that there's only one in 5 possible top-level rotations 
would result in C being ranked first: [C, D, E, A, B]. So the resulting order 
would be deterministic, but wouldn't distribute load evenly.
   
   Applying the affinity rotation "in post" does end up distributing load 
evenly, since you're always rotating mod the number of _equivalent_ replicas 
per group.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Solr or Lucene stickers?

[Alexandre Rafalovitch]

I have Solr one. I think there may have been a Lucene one as well.

No idea how to get them outside of ApacheCon though.

Regards,
 Alex

On Wed, Jul 10, 2019, 10:57 PM David Smiley, 
wrote:

> Does anyone know if Lucene or Solr stickers are made available anywhere,
> perhaps by the ASF?  Not only would I like some but some colleagues of mine
> inquired.
>
> ~ David Smiley
> Apache Lucene/Solr Search Developer
> http://www.linkedin.com/in/davidwsmiley
>

[GitHub] [lucene-solr] magibney commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

magibney commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302354143
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -197,6 +200,60 @@ private static boolean getDisableShardsWhitelist() {
 return Boolean.getBoolean(INIT_SOLR_DISABLE_SHARDS_WHITELIST);
   }
 
+  private static NamedList getNamedList(Object val) {
+if (val instanceof NamedList) {
+  return (NamedList)val;
+} else {
+  throw new IllegalArgumentException("Invalid config for replicaRouting: " 
+ val);
 
 Review comment:
   Agreed, will push update shortly.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] magibney commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

magibney commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302353887
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -197,6 +200,60 @@ private static boolean getDisableShardsWhitelist() {
 return Boolean.getBoolean(INIT_SOLR_DISABLE_SHARDS_WHITELIST);
   }
 
+  private static NamedList getNamedList(Object val) {
+if (val instanceof NamedList) {
+  return (NamedList)val;
+} else {
+  throw new IllegalArgumentException("Invalid config for replicaRouting: " 
+ val);
+}
+  }
+
+  private static String checkDefaultReplicaListTransformer(NamedList c, 
String setTo, String extantDefaultRouting) {
+if (!Boolean.TRUE.equals(c.getBooleanArg("default"))) {
+  return null;
+} else {
+  if (extantDefaultRouting == null) {
+return ShardParams.REPLICA_STABLE;
+  } else {
+throw new IllegalArgumentException("more than one routing scheme 
marked as default");
+  }
+}
+  }
+
+  private void initReplicaListTransformers(NamedList args) {
+String defaultRouting = null;
+if (args != null) {
+  NamedList routingConfig = (NamedList)args.get("replicaRouting");
 
 Review comment:
   Agreed, will push this change shortly. The error message is actually the 
main reason for the existence of the getNamedList() method, and the message 
definitely applies when used in this place.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] magibney commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

magibney commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302352238
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -449,9 +556,83 @@ private static boolean hasReplicaType(Object o, String 
preferred) {
 }
   }
 
+  private final ReplicaListTransformerFactory randomRltFactory = (String 
configSpec, SolrQueryRequest request,
+  ReplicaListTransformerFactory fallback) -> 
shufflingReplicaListTransformer;
+  private ReplicaListTransformerFactory stableRltFactory;
+  private ReplicaListTransformerFactory defaultRltFactory;
+
+  /**
+   * Private class responsible for applying pairwise sort based on inherent 
replica attributes,
+   * and subsequently reordering any equivalent replica sets according to 
behavior specified
+   * by the baseReplicaListTransformer.
+   */
+  private static final class TopLevelReplicaListTransformer implements 
ReplicaListTransformer {
+
+private final NodePreferenceRulesComparator replicaComp;
+private final ReplicaListTransformer baseReplicaListTransformer;
+
+public TopLevelReplicaListTransformer(NodePreferenceRulesComparator 
replicaComp, ReplicaListTransformer baseReplicaListTransformer) {
+  this.replicaComp = replicaComp;
+  this.baseReplicaListTransformer = baseReplicaListTransformer;
+}
+
+@Override
+public void transform(List choices) {
+  if (choices.size() > 1) {
+if (log.isDebugEnabled()) {
+  log.debug("Applying the following sorting preferences to replicas: 
{}",
+  Arrays.toString(replicaComp.preferenceRules.toArray()));
+}
+
+// First, sort according to comparator rules.
+try {
+  choices.sort(replicaComp);
+} catch (IllegalArgumentException iae) {
+  throw new SolrException(
+  SolrException.ErrorCode.BAD_REQUEST,
+  iae.getMessage()
 
 Review comment:
   This sounds reasonable to me, but this specific section of code is actually 
directly carried over from the existing code. I'm only mentioning this because 
I'm inclined to proceed with more caution here, in case this was a considered 
choice (for instance, if iae.getMessage() is expected to provide all the 
necessary information?). I notice that the IAEs that actually get thrown in 
NodePreferenceRulesComparator.compare() have pretty specific messages, so 
perhaps a the full exception/stack trace was deemed to be unnecessary? 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage

[Michael Gibney (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882628#comment-16882628
 ] 

Michael Gibney commented on SOLR-13257:
---

Thanks, [~cpoerschke] and [~tomasflobbe]! Responses shortly to PR comments.

[~cpoerschke], I think the examples (and answers) you set out are all 
illustrative and accurate (with the exception that the syntax would be 
{{shards.preference=replica.base:stable:dividend:fooBar=0}} or 
{{shards.preference=replica.base:stable:hash:fooBar=0)}}

A couple of comments on the initial example cases (though again it seems that 
the the answers you provide are accurate as far as I understand it):

2.Example: opt for possibility 2: It's a nitpicky semantic point, but the 
deterministic preference param doesn't ever really resolve directly to a 
particular replica, rather it resolves to a particular rotation of the replica 
preferences list. The first of these will in most (all?) cases be chosen to 
serve the request, but there's nothing special about the "top" replica _per 
se_, and thus nothing different about the _other_ replicas (that would cause 
them to be sorted differently).

4. "Does the new shard affinity logic influence the ordering within the end 
portion of the list too?"
Yes; each grouping of otherwise equivalent options is sorted and 
deterministically rotated according to the specified affinity param.

5. Same as 4; the new affinity logic only affects groupings of options that are 
otherwise (according to specified {{shards.preference}} param) considered to be 
equivalent; and it affects all such groupings, no matter how many hierarchical 
preferences are specified, or how many otherwise-equivalent groups there are.

> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch, 
> SOLR-13257.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request 
> randomly via 
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
>  to a particular replica. In setups with replication factor >1, this normally 
> results in a situation where subsequent requests (which one would hope/expect 
> to leverage cached results from previous related requests) end up getting 
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe 
> containing common terms?). The first request initializes the 
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and 
> there are a sufficient number of shards, subsequent requests will likely be 
> routed to at least one replica that _hasn't_ seen the query before. The 
> replicas with uninitialized caches become a bottleneck, and from the client's 
> perspective, many subsequent requests appear not to benefit from caching at 
> all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Solr or Lucene stickers?

[David Smiley]

Does anyone know if Lucene or Solr stickers are made available anywhere,
perhaps by the ASF?  Not only would I like some but some colleagues of mine
inquired.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley

[jira] [Commented] (LUCENE-8883) CHANGES.txt: Auto add issue categories on new releases

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882617#comment-16882617
 ] 

David Smiley commented on LUCENE-8883:
--

Optimization really does seam useful & meaningful, and I admit it's not easy to 
differentiate a New Feature from an Improvement.  

I do wish we also were in the habit of prefixing both CHANGES entries and 
commit messages with the module or feature in question, so that we/users 
needn't parse it out in our heads when reading them.

> CHANGES.txt: Auto add issue categories on new releases
> --
>
> Key: LUCENE-8883
> URL: https://issues.apache.org/jira/browse/LUCENE-8883
> Project: Lucene - Core
>  Issue Type: Task
>  Components: general/build
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Attachments: LUCENE-8883.patch, LUCENE-8883.patch
>
>
> As I write this, looking at Solr's CHANGES.txt for 8.2 I see we have some 
> sections: "Upgrade Notes", "New Features", "Bug Fixes", and "Other Changes".  
> There is no "Improvements" so no surprise here, the New Features category 
> has issues that ought to be listed as such.  I think the order vary as well.  
> I propose that on new releases, the initial state of the next release in 
> CHANGES.txt have these sections.  They can easily be removed at the upcoming 
> release if there are no such sections, or they could stay as empty.  It seems 
> addVersion.py is the code that sets this up and it could be enhanced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 8.2.0

[David Smiley]

BTW for 8.2.0 I updated Solr's CHANGES.txt to split out issues that seemed
to be Improvements that were not really New Features.

~ David Smiley
Apache Lucene/Solr Search Developer
http://www.linkedin.com/in/davidwsmiley


On Wed, Jul 10, 2019 at 10:38 AM Ignacio Vera  wrote:

> Thanks Tomoko for taking care of that.
>
> On Wed, Jul 10, 2019 at 4:03 PM Đạt Cao Mạnh 
> wrote:
>
>> Hi Ignacio,
>>
>> 8.1.2 bugfix release will cancelled. You can go ahead with 8.2 release.
>>
>> Thanks!
>>
>> On Wed, 10 Jul 2019 at 20:38, Tomoko Uchida 
>> wrote:
>>
>>> Hi,
>>> I opened a blocker issue a while ago for release 8.2:
>>> https://issues.apache.org/jira/browse/LUCENE-8907
>>>
>>> Sorry about that, I noticed the backwards incompatibility we have to
>>> deal with today. If there are no objections, I will revert the all
>>> related commits from the branch_8x and 8_2 in a few days.
>>>
>>> Thanks,
>>> Tomoko
>>>
>>> 2019年7月10日(水) 22:02 Ignacio Vera :
>>> >
>>> > Hi,
>>> >
>>> > All the issues listed above has been already committed and I see no
>>> blockers for release 8.2. I will cut the branch tomorrow around 10am CEST
>>> and I will wait for the decision on the bug release 8.1.2 to schedule the
>>> build of the first release candidate. Please let us know if this is
>>> troublesome for you.
>>> >
>>> > Thanks,
>>> >
>>> > Ignacio
>>> >
>>> >
>>> > On Tue, Jul 2, 2019 at 2:59 AM Joel Bernstein 
>>> wrote:
>>> >>
>>> >> I've got one issue that I'd like to get in (
>>> https://issues.apache.org/jira/browse/SOLR-13589), which I should have
>>> wrapped up in a day or two. +1 for around July 10th.
>>> >>
>>> >> On Mon, Jul 1, 2019 at 5:14 PM Nicholas Knize 
>>> wrote:
>>> >>>
>>> >>> +1 for starting the 8.2 release process. I think it would be good to
>>> get the LUCENE-8632 feature into 8.2 along with the BKD improvements and
>>> changes in LUCENE- and LUCENE-8896
>>> >>>
>>> >>> Nicholas Knize, Ph.D., GISP
>>> >>> Geospatial Software Guy  |  Elasticsearch
>>> >>> Apache Lucene PMC Member and Committer
>>> >>> nkn...@apache.org
>>> >>>
>>> >>>
>>> >>> On Wed, Jun 26, 2019 at 9:34 AM Ignacio Vera 
>>> wrote:
>>> 
>>>  Hi all,
>>> 
>>>  8.1 has been released on May 16th and we have new features,
>>> enhancements and fixes that are not released yet so I'd like to start
>>> thinking in releasing Lucene/Solr 8.2.0.
>>> 
>>>  I can create the 8.2 branch in two weeks time (around July 10th)
>>> and build the first RC by the end of that week if that works for everyone.
>>> Please let me know if there are bug fixes that needs to be fixed in 8.2 and
>>> might not be ready by then.
>>> 
>>>  Cheers,
>>> 
>>>  Ignacio
>>>
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>>
>>> --
>> *Best regards,*
>> *Cao Mạnh Đạt*
>> *E-mail: caomanhdat...@gmail.com *
>>
>

[jira] [Updated] (LUCENE-8883) CHANGES.txt: Auto add issue categories on new releases

[David Smiley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Smiley updated LUCENE-8883:
-
Attachment: LUCENE-8883.patch
Status: Open  (was: Open)

Nice unix piping :)

In this updated patch I added the "-" line beneath the 
headings because I reconsidered and found it pleasant on the eyes.  And I added 
the "(No changes)" below that for each because you asked.

For a bug fix release, the only heading is "Bug Fixes" (both Lucene & Solr),

Lucene's heading list:

['API Changes', 'New Features', 'Improvements', 'Bug Fixes', 'Other']

Solr's:

['Upgrade Notes', 'New Features', 'Improvements', 'Bug Fixes', 'Other Changes']

> CHANGES.txt: Auto add issue categories on new releases
> --
>
> Key: LUCENE-8883
> URL: https://issues.apache.org/jira/browse/LUCENE-8883
> Project: Lucene - Core
>  Issue Type: Task
>  Components: general/build
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Minor
> Attachments: LUCENE-8883.patch, LUCENE-8883.patch
>
>
> As I write this, looking at Solr's CHANGES.txt for 8.2 I see we have some 
> sections: "Upgrade Notes", "New Features", "Bug Fixes", and "Other Changes".  
> There is no "Improvements" so no surprise here, the New Features category 
> has issues that ought to be listed as such.  I think the order vary as well.  
> I propose that on new releases, the initial state of the next release in 
> CHANGES.txt have these sections.  They can easily be removed at the upcoming 
> release if there are no such sections, or they could stay as empty.  It seems 
> addVersion.py is the code that sets this up and it could be enhanced.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10377) Improve readability of the explain output for JSON format

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882580#comment-16882580
 ] 

David Smiley commented on SOLR-10377:
-

I really don't think everyone would like structured=true by default; I'm "meh" 
at best.  It's bulky.  I dunno if there's a good solution here.  I wonder if 
maybe when the admin UI renders the JSON, it might insert literal carriage 
returns after the occurrences of "\n" (actual backslash then 'n').  Or maybe 
somehow we could insert a pop-up with it rendered correctly.  Shrug... that's 
the best I can think up right now.  One way to ameliorate the situation is to 
add debug.explain.structured=true to the admin UI so you could click it easily 
and not have to know it exists.  Maybe that's the best we can do that also 
doesn't feel like a hack.

> Improve readability of the explain output for JSON format
> -
>
> Key: SOLR-10377
> URL: https://issues.apache.org/jira/browse/SOLR-10377
> Project: Solr
>  Issue Type: Improvement
>Reporter: Varun Thacker
>Priority: Minor
>
> Today when I ask solr for the debug query output In json with indent I get 
> this:
> {code}
> 1: " 3.545981 = sum of: 3.545981 = weight(name:dns in 0) [SchemaSimilarity], 
> result of: 3.545981 = score(doc=0,freq=1.0 = termFreq=1.0 ), product of: 
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 
> 0.5)) from: 2.0 = docFreq 24.0 = docCount 1.54 = tfNorm, computed as (freq * 
> (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 
> 1.0 = termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 
> 1.0 = fieldLength ",
> 2: " 7.4202514 = sum of: 7.4202514 = sum of: 2.7921112 = weight(name:domain 
> in 1) [SchemaSimilarity], result of: 2.7921112 = score(doc=1,freq=1.0 = 
> termFreq=1.0 ), product of: 2.3025851 = idf, computed as log(1 + (docCount - 
> docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 24.0 = docCount 
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * 
> fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 = parameter k1 
> 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 2.7921112 = 
> weight(name:name in 1) [SchemaSimilarity], result of: 2.7921112 = 
> score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 2.3025851 = idf, computed 
> as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 
> 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + 
> k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 
> = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of: 1.8360289 
> = score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 1.5141277 = idf, 
> computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 5.0 = 
> docFreq 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / 
> (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = 
> termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = 
> fieldLength "
> {code}
> When I run the same query with "wt=ruby" I get a much nicer output
> {code}
> '2'=>'
> 7.4202514 = sum of:
>   7.4202514 = sum of:
> 2.7921112 = weight(name:domain in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 2.7921112 = weight(name:name in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of:
>   1.8360289 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 1.5141277 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   5.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 

[jira] [Commented] (LUCENE-8366) upgrade to icu 62.1

[Robert Muir (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882575#comment-16882575
 ] 

Robert Muir commented on LUCENE-8366:
-

good catch! do you want to submit a fix?

> upgrade to icu 62.1
> ---
>
> Key: LUCENE-8366
> URL: https://issues.apache.org/jira/browse/LUCENE-8366
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/analysis
>Reporter: Robert Muir
>Priority: Major
> Fix For: trunk, 7.5
>
> Attachments: LUCENE-8366.patch
>
>
> This gives unicode 11 support.
> Also emoji tokenization is simpler and it gives a way to have better 
> tokenization for emoji from the future.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13240) UTILIZENODE action results in an exception

[Lucene/Solr QA (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882557#comment-16882557
 ] 

Lucene/Solr QA commented on SOLR-13240:
---

| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green}  0m 
 0s{color} | {color:green} The patch appears to include 1 new or modified test 
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  2m 
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green}  1m 
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} | 
{color:green}  1m 32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} | 
{color:green}  1m 32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} | 
{color:green}  1m 32s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}  5m  1s{color} 
| {color:red} solrj in the patch failed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 12m 47s{color} | 
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | solr.client.solrj.cloud.autoscaling.TestPolicyOld |
|   | solr.client.solrj.cloud.autoscaling.TestPolicy |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13240 |
| JIRA Patch URL | 
https://issues.apache.org/jira/secure/attachment/12974177/SOLR-13240.patch |
| Optional Tests |  compile  javac  unit  ratsources  checkforbiddenapis  
validatesourcepatterns  |
| uname | Linux lucene1-us-west 4.4.0-137-generic #163~14.04.1-Ubuntu SMP Mon 
Sep 24 17:14:57 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality | 
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
 |
| git revision | master / 8a277ca |
| ant | version: Apache Ant(TM) version 1.9.3 compiled on July 24 2018 |
| Default Java | LTS |
| unit | 
https://builds.apache.org/job/PreCommit-SOLR-Build/486/artifact/out/patch-unit-solr_solrj.txt
 |
|  Test Results | 
https://builds.apache.org/job/PreCommit-SOLR-Build/486/testReport/ |
| modules | C: solr/solrj U: solr/solrj |
| Console output | 
https://builds.apache.org/job/PreCommit-SOLR-Build/486/console |
| Powered by | Apache Yetus 0.7.0   http://yetus.apache.org |


This message was automatically generated.



> UTILIZENODE action results in an exception
> --
>
> Key: SOLR-13240
> URL: https://issues.apache.org/jira/browse/SOLR-13240
> Project: Solr
>  Issue Type: Bug
>  Components: AutoScaling
>Affects Versions: 7.6
>Reporter: Hendrik Haddorp
>Priority: Major
> Attachments: SOLR-13240.patch, SOLR-13240.patch, SOLR-13240.patch, 
> solr-solrj-7.5.0.jar
>
>
> When I invoke the UTILIZENODE action the REST call fails like this after it 
> moved a few replicas:
> {
>   "responseHeader":{
> "status":500,
> "QTime":40220},
>   "Operation utilizenode caused 
> exception:":"java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
>  Comparison method violates its general contract!",
>   "exception":{
> "msg":"Comparison method violates its general contract!",
> "rspCode":-1},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"Comparison method violates its general contract!",
> "trace":"org.apache.solr.common.SolrException: Comparison method violates 
> its general contract!\n\tat 
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
>  
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
>  
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
>  org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat 
> 

Re: New feature idea - Backwards (FST) dictionary for approximate string search

[Michael Sokolov]

You could create an alternate field in which you index the reverse tokens,
using a reverse token filter appended to the analysis chain of the main
field.

That leaves the question of how do you manage that in a service context,
and what kind of query makes use of two fields?

Otherwise you have to make a lower level change of some sort. I'm not sure
where you would do that cleanly.

On Wed, Jul 10, 2019, 7:40 AM Juan Caicedo 
wrote:

> I didn’t integrate it directly with a Lucene index. I used the FST class
> (and related utilities) to build a stand-alone spellchecker.
>
> However, the FST for the term dictionary is modified at index time only.
> At query time, we need to create a variation of the Levemshtein automata
> and match it with the FST accordingly.
>
> I need to take a closer look at the Lucene code to see how can we
> integrate it with the existing modules. I’ll do that this week.
>
> On Tue, Jul 9, 2019 at 03:18 Michael McCandless 
> wrote:
>
>> This sounds interesting!
>>
>> Did your patch build the FST at index time, so at search time all that
>> was needed was to load the FST from the index directory?
>>
>> Is the FST per-segment?
>>
>> Mike McCandless
>>
>> http://blog.mikemccandless.com
>>
>>
>> On Tue, Jul 9, 2019 at 1:43 AM Juan Caicedo 
>> wrote:
>>
>>> Hi Michael,
>>>
>>> I guess that I should have added more details :-)
>>>
>>> The main benefit from the technique is to do approximate search in
>>> large dictionaries. That is: find all the entries that are within 1 or
>>> 2 edit steps from the query. It's more useful for longer (starting
>>> from ~7 characters, iirc), but in general the technique can be applied
>>> to queries of any length. The main requirement is that we build the
>>> dictionary as an FST, using the backwards (reversed) keys of the
>>> original dictionary.
>>>
>>> I initially used the technique to implement a stand-alone
>>> spellchecker, but I think that it can also be used to optimize the
>>> Lucene fuzzy queries (e.g. for the spelling/suggest module). However,
>>> I'll need to look how can it be integrated with the part that creates
>>> the dictionary.
>>>
>>> I'll take a look at the code this week and I'll try to publish it in a
>>> public repository so that we can discuss about this with more concrete
>>> details.
>>>
>>> I skimmed the paper trying to understand possible applications of the
>>> technique. It sounds like efficient approximate (ie with some edits)
>>> substring search is the main idea? I don't believe such a query exists
>>> today in Lucene (nor any Suggester as far as I know). It sounds as if
>>> this would be useful for searching within large strings, eg DNA
>>> sequences or something like that, and maybe less applicable to typical
>>> "full text" (ie tokenized) search where the strings being searched are
>>> relatively shorter - does that sound right?
>>>
>>> On Sat, Jul 6, 2019 at 2:01 PM Michael Sokolov 
>>> wrote:
>>> >
>>> > Juan, that sounds intriguing.
>>> >
>>> > I skimmed the paper trying to understand possible applications of the
>>> > technique. It sounds like efficient approximate (ie with some edits)
>>> > substring search is the main idea? I don't believe such a query exists
>>> > today in Lucene (nor any Suggester as far as I know). It sounds as if
>>> > this would be useful for searching within large strings, eg DNA
>>> > sequences or something like that, and maybe less applicable to typical
>>> > "full text" (ie tokenized) search where the strings being searched are
>>> > relatively shorter - does that sound right?
>>> >
>>> > On Sat, Jul 6, 2019 at 12:35 PM Juan Caicedo <
>>> juan.caic...@cavorite.com> wrote:
>>> > >
>>> > > Hello,
>>> > >
>>> > > I've been working on a project for extending LevenshteinAutomata and
>>> > > I'd like to know if it would be useful to add it to Lucene.
>>> > >
>>> > > I've implemented the 'backwards dictionary' technique (see [1],
>>> > > section 6) for speeding up approximate search. This technique allows
>>> > > us to narrow down the search and, therefore, reduce the running time
>>> > > (at the expense of using more memory).
>>> > >
>>> > > I implemented it quite some time ago using an older version of
>>> Lucene,
>>> > > so I need to revisit the code. However, the implementation was
>>> > > relatively simple and it didn't require major changes to the core
>>> > > classes. I can share the code in a public repository and iterate on
>>> > > it, while I make it compatible for new Lucene APIs, add benchmarks,
>>> > > and more unit tests.
>>> > >
>>> > > Ideally, I'd like to contribute to Lucene, either as part of core,
>>> > > suggest or a different module.
>>> > >
>>> > > What do you think?
>>> > >
>>> > > [1]
>>> https://www.cis.uni-muenchen.de/download/publikationen/fastapproxsearch.pdf
>>> > >
>>> > > -
>>> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>>> > > For additional commands, e-mail: 

[jira] [Commented] (SOLR-13480) Collection creation failure when using Kerberos authentication combined with rule-base authorization

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882554#comment-16882554
 ] 

Ishan Chattopadhyaya commented on SOLR-13480:
-

[~moshebla], Are you using Autoscaling? If so, can you please attach the 
autoscaling.json for this?

> Collection creation failure when using Kerberos authentication combined with 
> rule-base authorization
> 
>
> Key: SOLR-13480
> URL: https://issues.apache.org/jira/browse/SOLR-13480
> Project: Solr
>  Issue Type: Bug
>  Components: Authorization, security
>Affects Versions: 7.7.1
>Reporter: mosh
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>  Labels: kerberos
>
> Creation of collection with an authorized user fails with the following error:
> {code:java}
> org.apache.solr.common.SolrException: Error getting replica locations : 
> unable to get autoscaling policy session{code}
> At first it may seem like SOLR-13355 duplication as we are using “all” 
> permission, but bug is specific to Kerberos (tested and found ok using basic 
> auth) plus we verified the failure with 7.7.2 snapshot that included the 
> relevant patch.
> +How to reproduce:+
> 1. Configure solr cloud with kerberos authentication and rule-based 
> authorization plugins using the following security.json file:
> {code:java}
> {
> "authentication":{
>    "class":"org.apache.solr.security.KerberosPlugin"
> },
> "authorization":{
>    "class":"solr.RuleBasedAuthorizationPlugin",
>    "permissions":[
>  {
>    "name":"read",
>    "role":"*"
>  },
>  {
>    "name":"all",
>    "role":"admin_user"
>  }
>    ],
>    "user-role":{
>  "admin_user@OUR_REALM":"admin_user"
>    }
> }}{code}
> 2. Create collection using an authorized user:
> {code:java}
> kinit admin_user@OUR_REALM
> curl --negotiate -u : 
> "http:///solr/admin/collections?action=CREATE=mycoll=1=_default"{code}
> {color:#d04437}==> request fails with the error written above.{color}
> 3. Disable authorization by removing _authorization_ section from 
> security.json, so file should be as follow:
> {code:java}
> {
>   "authentication":{
>     "class":"org.apache.solr.security.KerberosPlugin"
>   }
> }{code}
> 4. Create collection again as in step 2.
> {color:#14892c}==> request succeeds.{color}
> 5. Return authorization section to security.json (file from step 1) and make 
> sure authorization works as expected by inserting documents and executing 
> search queries with different users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-13472) HTTP requests to a node that does not hold a core of the collection are unauthorized

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16879967#comment-16879967
 ] 

Ishan Chattopadhyaya edited comment on SOLR-13472 at 7/11/19 12:55 AM:
---

Attaching a patch. As per this, the authorization should only be performed on a 
node if the request is to be executed on that node. Skips authorization on a 
node if the request has to be forwarded to another node. Note, 
forwardCredentials here is not necessary with this approach as PKI will pass on 
the original user principal as a payload for authorization on the forwarded 
node.

Doing this ^ should take care of the BasicAuth case here. However, for Kerberos 
case, the internode request is performed using Kerberos (not PKI), and hence 
the receiving node will not have the original user principal in order to 
authorize this request. I'll try to tackle this in -a separate JIRA- SOLR-13619.


was (Author: ichattopadhyaya):
Attaching a patch. As per this, the authorization should only be performed on a 
node if the request is to be executed on that node. Skips authorization on a 
node if the request has to be forwarded to another node. Note, 
forwardCredentials here is not necessary with this approach as PKI will pass on 
the original user principal as a payload for authorization on the forwarded 
node.

Doing this ^ should take care of the BasicAuth case here. However, for Kerberos 
case, the internode request is performed using Kerberos (not PKI), and hence 
the receiving node will not have the original user principal in order to 
authorize this request. I'll try to tackle this in a separate JIRA.

> HTTP requests to a node that does not hold a core of the collection are 
> unauthorized
> 
>
> Key: SOLR-13472
> URL: https://issues.apache.org/jira/browse/SOLR-13472
> Project: Solr
>  Issue Type: Bug
>  Components: Authorization
>Affects Versions: 7.7.1, 8.0
>Reporter: adfel
>Assignee: Ishan Chattopadhyaya
>Priority: Minor
>  Labels: security
> Attachments: SOLR-13472.patch, SOLR-13472.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> When creating collection in SolrCloud, collection is available for queries 
> and updates through all Solr nodes, in particular nodes that does not hold 
> one of collection's cores. This is expected behaviour that works when using 
> SolrJ client or HTTP requests.
> When enabling authorization rules it seems that this behaviour is broken for 
> HTTP requests:
>  - executing request to a node that holds part of the collection (core) obey 
> to authorization rules as expected.
>  - other nodes respond with code 403 - unauthorized request.
> SolrJ still works as expected.
> Tested both with BasicAuthPlugin and KerberosPlugin authentication plugins.
> +Steps for reproduce:+
> 1. Create a cloud made of 2 nodes (node_1, node_2).
> 2. Configure authentication and authorization by uploading following 
> security.json file to zookeeper:
>  
> {code:java}
> {
>  "authentication": {
>"blockUnknown": true,
>"class": "solr.BasicAuthPlugin",
>"credentials": {
>  "solr": "'solr' user password_hash",
>  "indexer_app": "'indexer_app' password_hash",
>  "read_user": "'read_user' password_hash"
>}
>  },
>  "authorization": {
>"class": "solr.RuleBasedAuthorizationPlugin",
>"permissions": [
>  {
>"name": "read",
>"role": "*"
>  },
>  {
>"name": "update",
>"role": [
>  "indexer",
>  "admin"
>]
>  },
>  {
>"name": "all",
>"role": "admin"
>  }
>],
>"user-role": {
>  "solr": "admin",
>  "indexer_app": "indexer"
>}
>  }
> }{code}
>  
> 3. create 'test' collection with one shard on *node_1*.
> -- 
> The following requests expected to succeed but return 403 status 
> (unauthorized request):
> {code:java}
> curl -u read_user:read_user "http://node_2/solr/test/select?q=*:*;
> curl -u indexer_app:indexer_app "http://node_2/solr/test/select?q=*:*;
> curl -u indexer_app:indexer_app "http://node_2/solr/test/update?commit=true;
> {code}
>  
> Authenticated '_solr_' user requests works as expected. My guess is due to 
> the special '_all_' role.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13619) Kerberos: 403 when node doesn't host collection

[Ishan Chattopadhyaya (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882548#comment-16882548
 ] 

Ishan Chattopadhyaya commented on SOLR-13619:
-

Internode communication also happens using Kerberos. Fix for this requires 
original user principal to be sent along, so that authorization can be skipped 
and done on the forwarded node using the original user principal (instead of 
the Solr node's service principal).

Added PR for this, https://github.com/apache/lucene-solr/pull/773.
This should be applied after applying SOLR-13472 fix. This is currently for 8x 
branch, will update for master at the time of committing.
[~noble.paul], can you please review?

> Kerberos: 403 when node doesn't host collection
> ---
>
> Key: SOLR-13619
> URL: https://issues.apache.org/jira/browse/SOLR-13619
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Ishan Chattopadhyaya
>Assignee: Ishan Chattopadhyaya
>Priority: Major
>
> This is a spin off from SOLR-13472, specifically to tackle the Kerberos case. 
> Here's the security.json to reproduce the same problem as of SOLR-13472:
> {code}
> {
>  "authentication": {"class": "org.apache.solr.security.KerberosPlugin"},
>  "authorization": {
>"class": "solr.RuleBasedAuthorizationPlugin",
>"permissions": [
>  {
>"name": "read",
>"role": "*"
>  },
>  {
>"name": "update",
>"role": [
>  "indexer",
>  "admin"
>]
>  },
>  {
>"name": "all",
>"role": "admin"
>  }
>],
>"user-role": {
>  "HTTP/so...@example.com": "admin",
>  "HTTP/so...@example.com": "admin",
>  "cli...@example.com": "indexer"
>}
>  }
> }
> {code}
> Here, cli...@example.com should be able to issue /update and /select requests 
> to both solr1 and solr2, but it throws 403 for the node that doesn't host the 
> collection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] chatman opened a new pull request #773: Kerberos: 403 when node doesn't host collection

[GitBox]

chatman opened a new pull request #773: Kerberos: 403 when node doesn't host 
collection
URL: https://github.com/apache/lucene-solr/pull/773
 
 
   
   
   
   # Description
   Kerberos: 403 when node doesn't host collection
   
   # Solution
   Forwarded requests with Kerberos should carry forward the original user 
principal.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882545#comment-16882545
 ] 

Tomoko Uchida commented on LUCENE-8907:
---

I don't want to stop 8.2 release for this issue.
If there is no strong objection within 24 hours, let me revert the changes 
(mentioned in my previous comment) for now and open another issue to discuss 
about backporting LUCENE-8778 to 8.x. It would be easy to cherry-pick those 
changes to the 8x branch again.

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
> Fix For: 8.2
>
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13619) Kerberos: 403 when node doesn't host collection

[Ishan Chattopadhyaya (JIRA)]

Ishan Chattopadhyaya created SOLR-13619:
---

 Summary: Kerberos: 403 when node doesn't host collection
 Key: SOLR-13619
 URL: https://issues.apache.org/jira/browse/SOLR-13619
 Project: Solr
  Issue Type: Improvement
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Ishan Chattopadhyaya
Assignee: Ishan Chattopadhyaya


This is a spin off from SOLR-13472, specifically to tackle the Kerberos case. 
Here's the security.json to reproduce the same problem as of SOLR-13472:
{code}
{
 "authentication": {"class": "org.apache.solr.security.KerberosPlugin"},
 "authorization": {
   "class": "solr.RuleBasedAuthorizationPlugin",
   "permissions": [
 {
   "name": "read",
   "role": "*"
 },
 {
   "name": "update",
   "role": [
 "indexer",
 "admin"
   ]
 },
 {
   "name": "all",
   "role": "admin"
 }
   ],
   "user-role": {
 "HTTP/so...@example.com": "admin",
 "HTTP/so...@example.com": "admin",
 "cli...@example.com": "indexer"
   }
 }
}
{code}

Here, cli...@example.com should be able to issue /update and /select requests 
to both solr1 and solr2, but it throws 403 for the node that doesn't host the 
collection.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1895 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1895/

3 tests failed.
FAILED:  
org.apache.lucene.codecs.memory.TestDirectDocValuesFormat.testSparseBooleanNumericsVsStoredFields

Error Message:
Test abandoned because suite timeout was reached.

Stack Trace:
java.lang.Exception: Test abandoned because suite timeout was reached.
at __randomizedtesting.SeedInfo.seed([66739E94873FF14C]:0)


FAILED:  
junit.framework.TestSuite.org.apache.lucene.codecs.memory.TestDirectDocValuesFormat

Error Message:
Suite timeout exceeded (>= 720 msec).

Stack Trace:
java.lang.Exception: Suite timeout exceeded (>= 720 msec).
at __randomizedtesting.SeedInfo.seed([66739E94873FF14C]:0)


FAILED:  
junit.framework.TestSuite.org.apache.solr.handler.TestReplicationHandler

Error Message:
ObjectTracker found 26 object(s) that were not released!!! 
[RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper, RawDirectoryWrapper, 
RawDirectoryWrapper, RawDirectoryWrapper] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.RawDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$fetchIndex$0(ReplicationHandler.java:345)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.RawDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$fetchIndex$0(ReplicationHandler.java:345)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.RawDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$fetchIndex$0(ReplicationHandler.java:345)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.RawDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$fetchIndex$0(ReplicationHandler.java:345)
  at java.base/java.lang.Thread.run(Thread.java:834)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.RawDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:509)  
at org.apache.solr.handler.IndexFetcher.fetchLatestIndex(IndexFetcher.java:351) 
 at 
org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:422) 
 at 
org.apache.solr.handler.ReplicationHandler.lambda$fetchIndex$0(ReplicationHandler.java:345)
  at java.base/java.lang.Thread.run(Thread.java:834)  

[jira] [Commented] (SOLR-13616) Possible racecondition/deadlock between collection DELETE and PrepRecovery ? (TestPolicyCloud failures)

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882523#comment-16882523
 ] 

ASF subversion and git services commented on SOLR-13616:


Commit 8a277cab7d15c03ff59577efceb6a0cb281d095a in lucene-solr's branch 
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=8a277ca ]

Harden TestPolicyCloud

- ensure all collections/replicas are active

- tighten assertions around expected replica locations

- eliminate some redundent code

These changes should also help ensure we don't get (more) spurious failures due 
to SOLR-13616


> Possible racecondition/deadlock between collection DELETE and PrepRecovery ? 
> (TestPolicyCloud failures)
> ---
>
> Key: SOLR-13616
> URL: https://issues.apache.org/jira/browse/SOLR-13616
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-13616.test-incomplete.patch, 
> thetaphi_Lucene-Solr-master-Linux_24358.log.txt
>
>
> Based on some recent jenkins failures in TestPolicyCloud, I suspect there is 
> a possible deadlock condition when attempting to delete a collection while 
> recovery is in progress.
> I haven't been able to identify exactly where/why/how the problem occurs, but 
> it does not appear to be a test specific problem, and seems like it could 
> potentially affect anyone unlucky enough to issue poorly timed DELETE.
> Details to follow in comments...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-12.0.1) - Build # 24382 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/24382/
Java: 64bit/jdk-12.0.1 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

3 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy

Error Message:
Timeout occurred while waiting response from server at: 
https://127.0.0.1:34429/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: Timeout occurred while 
waiting response from server at: https://127.0.0.1:34429/solr
at 
__randomizedtesting.SeedInfo.seed([176288B550DD7347:6E7895A1C6E8EFAF]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:667)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1128)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:897)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:829)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:228)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica(DeleteReplicaTest.java:384)
at 
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplicaLegacy(DeleteReplicaTest.java:264)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 

Re: 8.1.2 bug fix release

[Đạt Cao Mạnh]

Thanks Steve!

On Wed, 10 Jul 2019 at 22:35, Steve Rowe  wrote:

> I've disabled the ASF Jenkins 8.1 jobs.
>
> --
> Steve
>
> On Jul 10, 2019, at 9:39 AM, Đạt Cao Mạnh  wrote:
>
> Hi Adrien,
>
> Yeah, I am good with skipping/canceling 8.1.2 and focusing on 8.2 now.
>
> On Wed, 10 Jul 2019 at 19:11, Adrien Grand  wrote:
>
>> Hi Đạt,
>>
>> What do you think of focusing on 8.2 now, as Shalin suggested? Ignacio
>> had suggested cutting the 8.2 branch today initially so 8.2 could be
>> out pretty soon. Moreover if we want to move forward with 8.1.2, we
>> will likely have to delay 8.2 at this point.
>>
>> On Fri, Jul 5, 2019 at 8:42 AM Adrien Grand  wrote:
>> >
>> > Agreed with Shalin, we might want to focus on 8.2 at this point.
>> >
>> > On Fri, Jul 5, 2019 at 8:38 AM Shalin Shekhar Mangar
>> >  wrote:
>> > >
>> > > Thanks Dat.
>> > >
>> > > I don't think we should release a broken version without a fix for
>> SOLR-13413. A workaround for SOLR-13413 exists (forcing http1.1 for
>> inter-node requests) but we don't test that configuration anymore in Sole
>> so I am hesitant to suggest it.
>> > >
>> > > I think that either we agree to upgrade jetty to 9.4.19 in this point
>> release or we scrap it altogether and focus on 8.2.
>> > >
>> > > On Thu, Jul 4, 2019 at 4:54 PM Đạt Cao Mạnh 
>> wrote:
>> > >>
>> > >> Thanks Uwe!
>> > >>
>> > >> Hi guys, Ishan,
>> > >> When I tryied to build the RC1 for branch_8_1. I did see this
>> failure on test HttpPartitionWithTlogReplicasTest
>> > >>
>> > >> 215685 ERROR
>> (updateExecutor-537-thread-1-processing-x:collDoRecoveryOnRestart_shard1_replica_t1
>> r:core_node3 null n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart
>> s:shard1) [n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart
>> s:shard1 r:core_node3 x:collDoRecoveryOnRestart_shard1_replica_t1]
>> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
>> SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=StdNode:
>> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
>> to
>> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
>> > >>   => java.io.IOException: java.net.ConnectException:
>> Connection refused
>> > >> at
>> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
>> > >> java.io.IOException: java.net.ConnectException: Connection refused
>> > >> at
>> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
>> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
>> > >> at
>> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
>> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
>> > >> at
>> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
>> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
>> > >> at
>> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:169)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:337)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
>> ~[java/:?]
>> > >> at
>> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
>> ~[java/:?]
>> > >> at
>> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
>> ~[metrics-core-4.0.5.jar:4.0.5]
>> > >> at
>> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
>> ~[java/:?]
>> > >> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>> ~[?:1.8.0_191]
>> > >> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>> ~[?:1.8.0_191]
>> > >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
>> > >> Suppressed: java.io.IOException: java.net.ConnectException:
>> Connection refused
>> > >> at
>> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
>> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
>> > >> at
>> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
>> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
>> > >> at
>> 

[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882505#comment-16882505
 ] 

Tomás Fernández Löbbe commented on SOLR-13257:
--

This feature sounds great. I left some comments in the PR (not sure if the PR 
has the latest patch or not)

> Enable replica routing affinity for better cache usage
> --
>
> Key: SOLR-13257
> URL: https://issues.apache.org/jira/browse/SOLR-13257
> Project: Solr
>  Issue Type: New Feature
>  Components: SolrCloud
>Reporter: Michael Gibney
>Priority: Minor
> Attachments: AffinityShardHandlerFactory.java, SOLR-13257.patch, 
> SOLR-13257.patch
>
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> For each shard in a distributed request, Solr currently routes each request 
> randomly via 
> [ShufflingReplicaListTransformer|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/component/ShufflingReplicaListTransformer.java]
>  to a particular replica. In setups with replication factor >1, this normally 
> results in a situation where subsequent requests (which one would hope/expect 
> to leverage cached results from previous related requests) end up getting 
> routed to a replica that hasn't seen any related requests.
> The problem can be replicated by issuing a relatively expensive query (maybe 
> containing common terms?). The first request initializes the 
> {{queryResultCache}} on the consulted replicas. If replication factor >1 and 
> there are a sufficient number of shards, subsequent requests will likely be 
> routed to at least one replica that _hasn't_ seen the query before. The 
> replicas with uninitialized caches become a bottleneck, and from the client's 
> perspective, many subsequent requests appear not to benefit from caching at 
> all.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882500#comment-16882500
 ] 

Tomoko Uchida commented on LUCENE-8907:
---

Hi,
I picked the commits which should be reverted from commit log mail. There are 
five commits: all is mine and there is no conflicts other than 
lucene/CHANGES.txt on the reverted 8x branch. All tests is OK and that passed 
precommit.

{code}
git revert 2804d00137957dc27dce922103c6286bbc7a9eca
git revert 7e05bd717309088d3b35cba6be0f70fd14b3c8d0
git revert d1678a3a68685acd1432dfadcd5f49fa817273ad
git revert 12e3451fb80e151b16f51493761b9dfc580f8fa0
git revert 0cc1753e76eb61000c1d3513448719e5f7923e48
{code}

I think we can delay the discussion about backporting LUCENE-8778 into 8.x with 
transparent backwards compatibility. (From my perspective, the backporting to 
the 8x branch was my fault and we do not need to change the SPI loader until 
release 9.0.)

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
> Fix For: 8.2
>
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13617) 8.x memory issue

[Scott Yeadon (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882496#comment-16882496
 ] 

Scott Yeadon commented on SOLR-13617:
-

It hasn't been an issue from 4.x to 7.x, just from 8.x

If that's the case I guess I'll have to run 7.x until we do a server upgrade.

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-13617) 8.x memory issue

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882493#comment-16882493
 ] 

Jan Høydahl edited comment on SOLR-13617 at 7/10/19 10:39 PM:
--

Checked the attached logs. Appears your system has only 1Gb physical memory, 
which will always be too little to run both Ubuntu OS as well as Solr. Please 
provision a more powerful VM and try again!
{noformat}
Host: Intel(R) Xeon(R) CPU E5-2640 v4 @ 2.40GHz, 1 cores, 997M, Ubuntu 18.04.2 
LTS
{noformat}


was (Author: janhoy):
Checked the attached logs. Appears your system has only 1Gb physical memory, 
which will always be too little to run both Ubuntu OS as well as Solr. Please 
provision a more powerful VM and try again!

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13617) 8.x memory issue

[JIRA]

[ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882493#comment-16882493
 ] 

Jan Høydahl commented on SOLR-13617:


Checked the attached logs. Appears your system has only 1Gb physical memory, 
which will always be too little to run both Ubuntu OS as well as Solr. Please 
provision a more powerful VM and try again!

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] tflobbe commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

tflobbe commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302192407
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -449,9 +556,83 @@ private static boolean hasReplicaType(Object o, String 
preferred) {
 }
   }
 
+  private final ReplicaListTransformerFactory randomRltFactory = (String 
configSpec, SolrQueryRequest request,
+  ReplicaListTransformerFactory fallback) -> 
shufflingReplicaListTransformer;
+  private ReplicaListTransformerFactory stableRltFactory;
+  private ReplicaListTransformerFactory defaultRltFactory;
+
+  /**
+   * Private class responsible for applying pairwise sort based on inherent 
replica attributes,
+   * and subsequently reordering any equivalent replica sets according to 
behavior specified
+   * by the baseReplicaListTransformer.
+   */
+  private static final class TopLevelReplicaListTransformer implements 
ReplicaListTransformer {
+
+private final NodePreferenceRulesComparator replicaComp;
+private final ReplicaListTransformer baseReplicaListTransformer;
+
+public TopLevelReplicaListTransformer(NodePreferenceRulesComparator 
replicaComp, ReplicaListTransformer baseReplicaListTransformer) {
+  this.replicaComp = replicaComp;
+  this.baseReplicaListTransformer = baseReplicaListTransformer;
+}
+
+@Override
+public void transform(List choices) {
+  if (choices.size() > 1) {
+if (log.isDebugEnabled()) {
+  log.debug("Applying the following sorting preferences to replicas: 
{}",
+  Arrays.toString(replicaComp.preferenceRules.toArray()));
+}
+
+// First, sort according to comparator rules.
+try {
+  choices.sort(replicaComp);
+} catch (IllegalArgumentException iae) {
+  throw new SolrException(
+  SolrException.ErrorCode.BAD_REQUEST,
+  iae.getMessage()
 
 Review comment:
   May be useful to include the cause exception in the `SolrException`, since 
it could come from any comparator.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-13617) 8.x memory issue

[JIRA]

 [ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl resolved SOLR-13617.

Resolution: Invalid

Closing.

Now you have drawn attention to your initial email (sorry for no reply) so 
let’s continue discussion there.

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] tflobbe commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

tflobbe commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302282284
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -197,6 +200,60 @@ private static boolean getDisableShardsWhitelist() {
 return Boolean.getBoolean(INIT_SOLR_DISABLE_SHARDS_WHITELIST);
   }
 
+  private static NamedList getNamedList(Object val) {
+if (val instanceof NamedList) {
+  return (NamedList)val;
+} else {
+  throw new IllegalArgumentException("Invalid config for replicaRouting: " 
+ val);
 
 Review comment:
   I'd add here something like "Expecting NamedList but got..."


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] tflobbe commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

tflobbe commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302296475
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -449,9 +556,83 @@ private static boolean hasReplicaType(Object o, String 
preferred) {
 }
   }
 
+  private final ReplicaListTransformerFactory randomRltFactory = (String 
configSpec, SolrQueryRequest request,
+  ReplicaListTransformerFactory fallback) -> 
shufflingReplicaListTransformer;
+  private ReplicaListTransformerFactory stableRltFactory;
+  private ReplicaListTransformerFactory defaultRltFactory;
+
+  /**
+   * Private class responsible for applying pairwise sort based on inherent 
replica attributes,
+   * and subsequently reordering any equivalent replica sets according to 
behavior specified
+   * by the baseReplicaListTransformer.
+   */
+  private static final class TopLevelReplicaListTransformer implements 
ReplicaListTransformer {
+
+private final NodePreferenceRulesComparator replicaComp;
+private final ReplicaListTransformer baseReplicaListTransformer;
+
+public TopLevelReplicaListTransformer(NodePreferenceRulesComparator 
replicaComp, ReplicaListTransformer baseReplicaListTransformer) {
+  this.replicaComp = replicaComp;
+  this.baseReplicaListTransformer = baseReplicaListTransformer;
+}
+
+@Override
+public void transform(List choices) {
 
 Review comment:
   I may be very wrong here, but I think the output of this method would be the 
same (or at least the same in practice) if we first run the transformer on all 
the list (which essentially either shuffles it or uses the new affinity code to 
define the order) and then run the `choices.sort(replicaComp)`, since this last 
one is stable, and will maintain the order of equal elements.
   If that's the case, this method could be much simpler, and we don't need to 
find the boundaries, etc, plus the transformer gets called only once instead of 
once per group.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-SmokeRelease-8.x - Build # 145 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-8.x/145/

No tests ran.

Build Log:
[...truncated 24989 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2587 links (2117 relative) to 3396 anchors in 259 files
 [echo] Validated Links & Anchors via: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/solr-8.2.0.tgz
 into 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/x1/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings 

[GitHub] [lucene-solr] tflobbe commented on a change in pull request #677: SOLR-13257: support for stable replica routing preferences

[GitBox]

tflobbe commented on a change in pull request #677: SOLR-13257: support for 
stable replica routing preferences
URL: https://github.com/apache/lucene-solr/pull/677#discussion_r302281692
 
 

 ##
 File path: 
solr/core/src/java/org/apache/solr/handler/component/HttpShardHandlerFactory.java
 ##
 @@ -197,6 +200,60 @@ private static boolean getDisableShardsWhitelist() {
 return Boolean.getBoolean(INIT_SOLR_DISABLE_SHARDS_WHITELIST);
   }
 
+  private static NamedList getNamedList(Object val) {
+if (val instanceof NamedList) {
+  return (NamedList)val;
+} else {
+  throw new IllegalArgumentException("Invalid config for replicaRouting: " 
+ val);
+}
+  }
+
+  private static String checkDefaultReplicaListTransformer(NamedList c, 
String setTo, String extantDefaultRouting) {
+if (!Boolean.TRUE.equals(c.getBooleanArg("default"))) {
+  return null;
+} else {
+  if (extantDefaultRouting == null) {
+return ShardParams.REPLICA_STABLE;
+  } else {
+throw new IllegalArgumentException("more than one routing scheme 
marked as default");
+  }
+}
+  }
+
+  private void initReplicaListTransformers(NamedList args) {
+String defaultRouting = null;
+if (args != null) {
+  NamedList routingConfig = (NamedList)args.get("replicaRouting");
 
 Review comment:
   You could also use your `getNamedList` method here for a better error 
message at the top level of the config (after a null check)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[JIRA]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882483#comment-16882483
 ] 

Jan Høydahl commented on LUCENE-8907:
-

Reverting sounds to cause least friction, not delaying 8.2. Unless a patch is 
presented within say 24h, which is very simple and straight forward.?

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
> Fix For: 8.2
>
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13617) 8.x memory issue

[Scott Yeadon (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882472#comment-16882472
 ] 

Scott Yeadon commented on SOLR-13617:
-

I did post to the solr-user mailing list, but got no response 
([http://lucene.472066.n3.nabble.com/8-0-upgrade-issue-td4440554.html)]

I've attached the console and pid log files.

 

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13617) 8.x memory issue

[Scott Yeadon (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Yeadon updated SOLR-13617:

Attachment: hs_err_pid17948.log.8.1.1
hs_console_log.8.1.1

> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
> Attachments: hs_console_log.8.1.1, hs_err_pid17948.log.8.1.1
>
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[JIRA]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jan Høydahl updated LUCENE-8907:

Fix Version/s: 8.2

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
> Fix For: 8.2
>
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-Linux (64bit/jdk-12.0.1) - Build # 856 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Linux/856/
Java: 64bit/jdk-12.0.1 -XX:+UseCompressedOops -XX:+UseG1GC

17 tests failed.
FAILED:  
org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat

Error Message:
expected:<[2]> but was:<[null]>

Stack Trace:
org.junit.ComparisonFailure: expected:<[2]> but was:<[null]>
at 
__randomizedtesting.SeedInfo.seed([223EC5E8550998AA:48EDEA0D2162B415]:0)
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.solr.cloud.CollectionsAPISolrJTest.testCreateCollWithDefaultClusterPropertiesOldFormat(CollectionsAPISolrJTest.java:155)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:835)


FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.raceConditionOnDeleteAndRegisterReplica

Error Message:
Timeout occurred while waiting response from server at: 
https://127.0.0.1:44491/solr

Stack Trace:

[jira] [Commented] (SOLR-13257) Enable replica routing affinity for better cache usage

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882405#comment-16882405
 ] 

Christine Poerschke commented on SOLR-13257:


Thanks [~mgibney] for creating and working on this ticket!

I have some "requirements" or "intended behaviour" type questions or thoughts, 
perhaps best posed via examples.

1. The out-of-the-box behaviour (in the absence of {{shards.preference}} or 
similar parameters) is, as you say, a random choice.
 * 
 ** Example: A collection with three replicas (replicaA, replicaB, replicaC), 
all of which are available at the time of searching.
 ** Expected behaviour: The list of available replicas is randomly shuffled 
i.e. there are {{3! = 6}} possibilities overall: {{[ [A, B, C], [A, C, B], [B, 
A, C], [B, C, A], [C, A, B], [C, B, A] ]}}

2. The objective of this ticket i.e. the proposed behaviour (in the absence of 
{{shards.preference}} or similar parameters) is reduce or remove the 
random-ness of the out-of-the-box behaviour.
 * 
 ** Example: A collection with three replicas (replicaA, replicaB, replicaC), 
all of which are available at the time of searching. The request indicates 
(directly via a numeric value parameter or indirectly via a non-numeric value 
parameter that can be hashed) the affinity to an element in or a portion of the 
list of replicas. Let's assume the affinity expressed translates into replica 
{{A}}.
 ** Expected behaviour:
 *** Possibility 1: The preferred replica is {{A}} but beyond the first choice 
the random-ness remains for load balancing purposes i.e. there are {{2! = 2}} 
possibilities overall: {{[ [A, B, C], [A, C, B] ]}}
 *** Possibility 2: The preferred replica is {{A}} and also beyond that first 
choice there is no random-ness i.e. the replica list is sorted 
deterministically to give just one possibility: {{[A, B, C]}}

3. The existing {{shards.preference}} parameter influences the replica choices 
(in the absence of new affinity logic).

(caveat: I presume the below is the existing behaviour of the existing code but 
have not recently read the code in detail to fully convince myself that this is 
so.)
 * 
 ** Example 1: A collection with six replicas spread equally across three 
locations. The request indicates a preference for "Europe" location.
 ** Expected behaviour: The preferred replicas (based on location) are at the 
start of the list and the not-preferred replicas are at the end of the list. 
The ordering within the start and end portions of the list is random giving 
{{2! * 4! = 2*1 * 4*3*2*1 = 2 * 24 = 48}} possibilities overall.

 * 
 ** Example 2: A collection with six replicas spread equally across three 
locations and the two replicas at each location are of different types e.g. 
{{PULL}} and {{TLOG}}. The request indicates a preference primarily for 
"Europe" location and secondarily for "PULL" type.
 ** Expected behaviour: The preferred replicas (based on location) are at the 
start of the list and the not-preferred replicas are at the end of the list. 
The ordering within the start portion of the list becomes deterministic via the 
secondary preference (e.g. {{[ replicaB1(europe,pull), replicaB2(europe,tlog) 
]}}) and the ordering within the end portion of the list is narrowed down via 
the secondary preference but some random-ness remains giving 4 possibilities 
overall:
{code:java}
[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaA1(america,pull), replicaC1(asia,pull),
  replicaA2(america,tlog), replicaC2(asia,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaA1(america,pull), replicaC1(asia,pull),
  replicaC2(asia,tlog), replicaA2(america,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaC1(asia,pull), replicaA1(america,pull),
  replicaC2(asia,tlog), replicaA2(america,tlog) ]

[ replicaB1(europe,pull), replicaB2(europe,tlog),
  replicaC1(asia,pull), replicaA1(america,pull),
  replicaA2(america,tlog), replicaC2(asia,tlog) ]
{code}

4. The new shard affinity logic combined with a single {{shards.preference}} 
parameter.
 * 
 ** Example: A collection with six replicas spread equally across three 
locations. The request indicates a preference for "Europe" location.
 ** Expected behaviour: The preferred replicas (based on location) are at the 
start of the list and the not-preferred replicas are at the end of the list.
 *** The ordering within the "Europe" start portion of the list becomes 
deterministic via the new shard affinity logic.
 *** Does the new shard affinity logic influence the ordering within the end 
portion of the list too?

5. The new shard affinity logic combined with multiple {{shards.preference}} 
parameters.
 * 
 ** At its simplest each {{shards.preference}} parameter will have two choices 
e.g. "PULL" vs. "TLOG" replica type or "Europe" vs. unknown replica location.
 ** Assuming two {{shards.preference}} parameters with two choices each the 

[jira] [Comment Edited] (SOLR-13399) compositeId support for shard splitting

[Yonik Seeley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13399?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882321#comment-16882321
 ] 

Yonik Seeley edited comment on SOLR-13399 at 7/10/19 6:43 PM:
--

Here's a draft patch (no tests yet) for feedback.
This adds a parameter "splitByPrefix" to SPLITSHARD.  When the overseer sees 
this parameter, it sends an additional SPLIT request with the "getRanges" 
parameter set.  This causes SPLIT (SplitOp.java) to calculate the ranges based 
on the prefix field "id_prefix" and return the recommended split string in the 
response in the "ranges" parameter.  SPLITSHARD in the overseer then proceeds 
as if that ranges string had been passed in by the user.

"id_prefix" is currently populated via a copyField in the schema:
{code}

  
  
  

  

  
{code}

The prefix field is currently always "id_prefix" (convention / implicit).  Not 
sure if it adds value to make it configurable via a "field" parameter on the 
SPLITSHARD command.



was (Author: ysee...@gmail.com):
Here's a draft patch (no tests yet) for feedback.
This adds a parameter "splitByPrefix" to SPLITSHARD.  When the overseer sees 
this parameter, it sends an additional SPLIT request with the "getRanges" 
parameter set.  This causes SPLIT (SplitOp.java) to calculate the ranges based 
on the prefix field "id_prefix" and return the recommended split string in the 
response in the "ranges" parameter.  SPLITSHARD in the overseer then proceeds 
as if that ranges string had been passed in by the user.

"id_prefix" is currently populated via a copyField in the schema:
{code}

  
  
  

  

  
{code}

The field "id_prefix" is currently hard-coded.  Perhaps this should be made 
configurable via a "field" parameter on the SPLITSHARD command?


> compositeId support for shard splitting
> ---
>
> Key: SOLR-13399
> URL: https://issues.apache.org/jira/browse/SOLR-13399
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Major
> Attachments: SOLR-13399.patch
>
>
> Shard splitting does not currently have a way to automatically take into 
> account the actual distribution (number of documents) in each hash bucket 
> created by using compositeId hashing.
> We should probably add a parameter *splitByPrefix* to the *SPLITSHARD* 
> command that would look at the number of docs sharing each compositeId prefix 
> and use that to create roughly equal sized buckets by document count rather 
> than just assuming an equal distribution across the entire hash range.
> Like normal shard splitting, we should bias against splitting within hash 
> buckets unless necessary (since that leads to larger query fanout.) . Perhaps 
> this warrants a parameter that would control how much of a size mismatch is 
> tolerable before resorting to splitting within a bucket. 
> *allowedSizeDifference*?
> To more quickly calculate the number of docs in each bucket, we could index 
> the prefix in a different field.  Iterating over the terms for this field 
> would quickly give us the number of docs in each (i.e lucene keeps track of 
> the doc count for each term already.)  Perhaps the implementation could be a 
> flag on the *id* field... something like *indexPrefixes* and poly-fields that 
> would cause the indexing to be automatically done and alleviate having to 
> pass in an additional field during indexing and during the call to 
> *SPLITSHARD*.  This whole part is an optimization though and could be split 
> off into its own issue if desired.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8909) Deprecate getFieldNames from IndexWriter

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882325#comment-16882325
 ] 

Adrien Grand commented on LUCENE-8909:
--

+1

> Deprecate getFieldNames from IndexWriter
> 
>
> Key: LUCENE-8909
> URL: https://issues.apache.org/jira/browse/LUCENE-8909
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Munendra S N
>Priority: Major
>
> From SOLR-12368
> {quote}Would be nice to be able to remove IndexWriter.getFieldNames as well, 
> which was added in LUCENE-7659 only for this workaround.{quote}
> Once Solr task resolved, deprecate {{IndexWriter#getFieldNames}} from 8x and 
> remove it from master



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13399) compositeId support for shard splitting

[Yonik Seeley (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yonik Seeley updated SOLR-13399:

Attachment: SOLR-13399.patch
Status: Open  (was: Open)

Here's a draft patch (no tests yet) for feedback.
This adds a parameter "splitByPrefix" to SPLITSHARD.  When the overseer sees 
this parameter, it sends an additional SPLIT request with the "getRanges" 
parameter set.  This causes SPLIT (SplitOp.java) to calculate the ranges based 
on the prefix field "id_prefix" and return the recommended split string in the 
response in the "ranges" parameter.  SPLITSHARD in the overseer then proceeds 
as if that ranges string had been passed in by the user.

"id_prefix" is currently populated via a copyField in the schema:
{code}

  
  
  

  

  
{code}

The field "id_prefix" is currently hard-coded.  Perhaps this should be made 
configurable via a "field" parameter on the SPLITSHARD command?


> compositeId support for shard splitting
> ---
>
> Key: SOLR-13399
> URL: https://issues.apache.org/jira/browse/SOLR-13399
> Project: Solr
>  Issue Type: New Feature
>Reporter: Yonik Seeley
>Priority: Major
> Attachments: SOLR-13399.patch
>
>
> Shard splitting does not currently have a way to automatically take into 
> account the actual distribution (number of documents) in each hash bucket 
> created by using compositeId hashing.
> We should probably add a parameter *splitByPrefix* to the *SPLITSHARD* 
> command that would look at the number of docs sharing each compositeId prefix 
> and use that to create roughly equal sized buckets by document count rather 
> than just assuming an equal distribution across the entire hash range.
> Like normal shard splitting, we should bias against splitting within hash 
> buckets unless necessary (since that leads to larger query fanout.) . Perhaps 
> this warrants a parameter that would control how much of a size mismatch is 
> tolerable before resorting to splitting within a bucket. 
> *allowedSizeDifference*?
> To more quickly calculate the number of docs in each bucket, we could index 
> the prefix in a different field.  Iterating over the terms for this field 
> would quickly give us the number of docs in each (i.e lucene keeps track of 
> the doc count for each term already.)  Perhaps the implementation could be a 
> flag on the *id* field... something like *indexPrefixes* and poly-fields that 
> would cause the indexing to be automatically done and alleviate having to 
> pass in an additional field during indexing and during the call to 
> *SPLITSHARD*.  This whole part is an optimization though and could be split 
> off into its own issue if desired.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12368) in-place DV updates should no longer have to jump through hoops if field does not yet exist

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882314#comment-16882314
 ] 

Munendra S N commented on SOLR-12368:
-

 [^SOLR-12368.patch] 
Patch without any changes in lucene

> in-place DV updates should no longer have to jump through hoops if field does 
> not yet exist
> ---
>
> Key: SOLR-12368
> URL: https://issues.apache.org/jira/browse/SOLR-12368
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12368.patch, SOLR-12368.patch, SOLR-12368.patch, 
> SOLR-12368.patch
>
>
> When SOLR-5944 first added "in-place" DocValue updates to Solr, one of the 
> edge cases thta had to be dealt with was the limitation imposed by 
> IndexWriter that docValues could only be updated if they already existed - if 
> a shard did not yet have a document w/a value in the field where the update 
> was attempted, we would get an error.
> LUCENE-8316 seems to have removed this error, which i believe means we can 
> simplify & speed up some of the checks in Solr, and support this situation as 
> well, rather then falling back on full "read stored fields & reindex" atomic 
> update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12368) in-place DV updates should no longer have to jump through hoops if field does not yet exist

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N updated SOLR-12368:

Attachment: SOLR-12368.patch

> in-place DV updates should no longer have to jump through hoops if field does 
> not yet exist
> ---
>
> Key: SOLR-12368
> URL: https://issues.apache.org/jira/browse/SOLR-12368
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12368.patch, SOLR-12368.patch, SOLR-12368.patch, 
> SOLR-12368.patch
>
>
> When SOLR-5944 first added "in-place" DocValue updates to Solr, one of the 
> edge cases thta had to be dealt with was the limitation imposed by 
> IndexWriter that docValues could only be updated if they already existed - if 
> a shard did not yet have a document w/a value in the field where the update 
> was attempted, we would get an error.
> LUCENE-8316 seems to have removed this error, which i believe means we can 
> simplify & speed up some of the checks in Solr, and support this situation as 
> well, rather then falling back on full "read stored fields & reindex" atomic 
> update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8909) Deprecate getFieldNames from IndexWriter

[Munendra S N (JIRA)]

Munendra S N created LUCENE-8909:


 Summary: Deprecate getFieldNames from IndexWriter
 Key: LUCENE-8909
 URL: https://issues.apache.org/jira/browse/LUCENE-8909
 Project: Lucene - Core
  Issue Type: Task
Reporter: Munendra S N


>From SOLR-12368
{quote}Would be nice to be able to remove IndexWriter.getFieldNames as well, 
which was added in LUCENE-7659 only for this workaround.{quote}

Once Solr task resolved, deprecate {{IndexWriter#getFieldNames}} from 8x and 
remove it from master



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Uwe Schindler (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882309#comment-16882309
 ] 

Uwe Schindler commented on LUCENE-8907:
---

I agree partly with Adrien. But we can still allow to use old factories in 8.2, 
in a lenient way: If the factory has no NAME field, it generates one using old 
algorithm - and ideally would log a warning. The biggest problem ist that we 
can't log warnings, which is not good. Solr could do this.

Sorry for coming up with this issue so late, but I did not notice that the 
whole change was committed to 8.x. I have here a customer project with many 
custom analyzers, so it came to my mind.

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] magibney edited a comment on issue #772: Payload-stored position length intervals

[GitBox]

magibney edited a comment on issue #772: Payload-stored position length 
intervals
URL: https://github.com/apache/lucene-solr/pull/772#issuecomment-510162543
 
 
   Would you want to consider reordering tokens in PayloadTokenLengthFilter, so 
that query implementations can expect a reliable ordering of tokens from the 
index? I think this would affect the order of terms recorded in the index only 
for "prefix phrases", as might be generated with a synonym such as 
"toaster"=="toaster oven"; but if we're recording position length in the index, 
it's good to have a defined ordering that reflects this new structural 
information.
   Reordering would be similar to that done by FlattenGraphFilter; for 
reference, an example that reorders for position length (and records position 
length to Payloads) can be found 
[here](https://github.com/magibney/lucene-solr/blob/LUCENE-7398/master/lucene/core/src/java/org/apache/lucene/analysis/core/PositionLengthOrderTokenFilter.java).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] magibney commented on issue #772: Payload-stored position length intervals

[GitBox]

magibney commented on issue #772: Payload-stored position length intervals
URL: https://github.com/apache/lucene-solr/pull/772#issuecomment-510162543
 
 
   Would you want to consider reordering tokens in PositionLengthTokenFilter, 
so that query implementations can expect a reliable ordering of tokens from the 
index? I think this would affect the order of terms recorded in the index only 
for "prefix phrases", as might be generated with a synonym such as 
"toaster"=="toaster oven"; but if we're recording position length in the index, 
it's good to have a defined ordering that reflects this new structural 
information.
   Reordering would be similar to that done by FlattenGraphFilter; for 
reference, an example that reorders for position length (and records position 
length to Payloads) can be found 
[here](https://github.com/magibney/lucene-solr/blob/LUCENE-7398/master/lucene/core/src/java/org/apache/lucene/analysis/core/PositionLengthOrderTokenFilter.java).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8875) Should TopScoreDocCollector Always Populate Sentinel Values?

[Adrien Grand (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Adrien Grand resolved LUCENE-8875.
--
   Resolution: Fixed
Fix Version/s: 8.2

> Should TopScoreDocCollector Always Populate Sentinel Values?
> 
>
> Key: LUCENE-8875
> URL: https://issues.apache.org/jira/browse/LUCENE-8875
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
> Fix For: 8.2
>
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> TopScoreDocCollector always initializes HitQueue as the PQ implementation, 
> and instruct HitQueue to populate with sentinels. While this is a great 
> safety mechanism, for very large datasets where the query's selectivity is 
> high, the sentinel population can be redundant and can become a large enough 
> bottleneck in itself. Does it make sense to introduce a new parameter in 
> TopScoreDocCollector which uses a heuristic (say number of hits > 10k) and 
> does not populate sentinels?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8875) Should TopScoreDocCollector Always Populate Sentinel Values?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882295#comment-16882295
 ] 

ASF subversion and git services commented on LUCENE-8875:
-

Commit 7339eb272c30e993e0a8e73154fdfca8ef9879e4 in lucene-solr's branch 
refs/heads/branch_8x from Atri Sharma
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=7339eb2 ]

LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits (#754)

This commit introduces a new collector which is optimized for
cases when the number of hits is large and/or the actual hits
collected are sparse in comparison to the number of hits
requested.

> Should TopScoreDocCollector Always Populate Sentinel Values?
> 
>
> Key: LUCENE-8875
> URL: https://issues.apache.org/jira/browse/LUCENE-8875
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> TopScoreDocCollector always initializes HitQueue as the PQ implementation, 
> and instruct HitQueue to populate with sentinels. While this is a great 
> safety mechanism, for very large datasets where the query's selectivity is 
> high, the sentinel population can be redundant and can become a large enough 
> bottleneck in itself. Does it make sense to introduce a new parameter in 
> TopScoreDocCollector which uses a heuristic (say number of hits > 10k) and 
> does not populate sentinels?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12368) in-place DV updates should no longer have to jump through hoops if field does not yet exist

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882292#comment-16882292
 ] 

Munendra S N commented on SOLR-12368:
-

Thanks for the review [~hossman]
I haven't yet created the patch with only solr changes. Also, will create 
related jira in lucene project

> in-place DV updates should no longer have to jump through hoops if field does 
> not yet exist
> ---
>
> Key: SOLR-12368
> URL: https://issues.apache.org/jira/browse/SOLR-12368
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12368.patch, SOLR-12368.patch, SOLR-12368.patch
>
>
> When SOLR-5944 first added "in-place" DocValue updates to Solr, one of the 
> edge cases thta had to be dealt with was the limitation imposed by 
> IndexWriter that docValues could only be updated if they already existed - if 
> a shard did not yet have a document w/a value in the field where the update 
> was attempted, we would get an error.
> LUCENE-8316 seems to have removed this error, which i believe means we can 
> simplify & speed up some of the checks in Solr, and support this situation as 
> well, rather then falling back on full "read stored fields & reindex" atomic 
> update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz commented on a change in pull request #772: Payload-stored position length intervals

[GitBox]

jpountz commented on a change in pull request #772: Payload-stored position 
length intervals
URL: https://github.com/apache/lucene-solr/pull/772#discussion_r302183512
 
 

 ##
 File path: 
lucene/sandbox/src/java/org/apache/lucene/payloads/PayloadTokenLengthFilter.java
 ##
 @@ -0,0 +1,71 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.payloads;
+
+import java.io.IOException;
+
+import org.apache.lucene.analysis.TokenFilter;
+import org.apache.lucene.analysis.TokenStream;
+import org.apache.lucene.analysis.tokenattributes.PayloadAttribute;
+import org.apache.lucene.analysis.tokenattributes.PositionLengthAttribute;
+import org.apache.lucene.util.BytesRef;
+
+/**
+ * Encodes a token's position length into its associated payload
+ *
+ * @see PayloadLengthTermIntervalsSource
+ */
+public final class PayloadTokenLengthFilter extends TokenFilter {
 
 Review comment:
   Create a factory for it as well?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz commented on a change in pull request #772: Payload-stored position length intervals

[GitBox]

jpountz commented on a change in pull request #772: Payload-stored position 
length intervals
URL: https://github.com/apache/lucene-solr/pull/772#discussion_r302179849
 
 

 ##
 File path: 
lucene/sandbox/src/java/org/apache/lucene/payloads/PayloadLengthTermIntervalsSource.java
 ##
 @@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.payloads;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Objects;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.index.Terms;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.queries.intervals.IntervalIterator;
+import org.apache.lucene.queries.intervals.IntervalQuery;
+import org.apache.lucene.queries.intervals.IntervalsSource;
+import org.apache.lucene.search.MatchesIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryVisitor;
+import org.apache.lucene.util.BytesRef;
+
+/**
+ * An IntervalsSource that returns intervals over single terms, reading token
+ * length from each term's associated payload
+ *
+ * Tokens should be indexed using {@link PayloadTokenLengthFilter} to write
+ * token lengths into payloads
+ */
+public class PayloadLengthTermIntervalsSource extends IntervalsSource {
+
+  private final BytesRef term;
+
+  /**
+   * Create a PayloadLengthTermIntervalsSource for the given term
+   */
+  public PayloadLengthTermIntervalsSource(BytesRef term) {
+this.term = term;
+  }
+
+  @Override
+  public IntervalIterator intervals(String field, LeafReaderContext ctx) 
throws IOException {
+Terms terms = ctx.reader().terms(field);
+if (terms == null)
+  return null;
+if (terms.hasPositions() == false) {
+  throw new IllegalArgumentException("Cannot create an IntervalIterator 
over field " + field + " because it has no indexed positions");
+}
+TermsEnum te = terms.iterator();
+if (te.seekExact(term) == false) {
+  return null;
+}
+return intervals(term, te);
+  }
+
+  private static int decodePayload(BytesRef payload) {
+if (payload == null) {
+  return 1;
+}
+int length = 0;
+for (int i = payload.offset; i < payload.offset + payload.length; i++) {
+  length = (length << 8) | Byte.toUnsignedInt(payload.bytes[i]);
+}
+return length;
+  }
+
+  private static IntervalIterator intervals(BytesRef term, TermsEnum te) 
throws IOException {
+PostingsEnum pe = te.postings(null, PostingsEnum.PAYLOADS);
+float cost = termPositionsCost(te);
+return new IntervalIterator() {
+
+  int length;
+
+  @Override
+  public int docID() {
+return pe.docID();
+  }
+
+  @Override
+  public int nextDoc() throws IOException {
+int doc = pe.nextDoc();
+reset();
+return doc;
+  }
+
+  @Override
+  public int advance(int target) throws IOException {
+int doc = pe.advance(target);
+reset();
+return doc;
+  }
+
+  @Override
+  public long cost() {
+return pe.cost();
+  }
+
+  int pos = -1, upto;
+
+  @Override
+  public int start() {
+return pos;
+  }
+
+  @Override
+  public int end() {
+return pos + length - 1;
+  }
+
+  @Override
+  public int gaps() {
+return 0;
+  }
+
+  @Override
+  public int nextInterval() throws IOException {
+if (upto <= 0)
+  return pos = NO_MORE_INTERVALS;
+upto--;
+pos = pe.nextPosition();
+length = decodePayload(pe.getPayload());
+return pos;
+  }
+
+  @Override
+  public float matchCost() {
+return cost;
+  }
+
+  private void reset() throws IOException {
+if (pe.docID() == NO_MORE_DOCS) {
+  upto = -1;
+  pos = NO_MORE_INTERVALS;
+}
+else {
+  upto = pe.freq();
+  pos = -1;
+}
+  }
+
+  @Override
+  public String toString() {
+return term.utf8ToString() + ":" + super.toString();
+  }
+};
+  

[GitHub] [lucene-solr] jpountz commented on a change in pull request #772: Payload-stored position length intervals

[GitBox]

jpountz commented on a change in pull request #772: Payload-stored position 
length intervals
URL: https://github.com/apache/lucene-solr/pull/772#discussion_r302183057
 
 

 ##
 File path: 
lucene/sandbox/src/java/org/apache/lucene/payloads/PayloadLengthTermIntervalsSource.java
 ##
 @@ -0,0 +1,299 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.lucene.payloads;
+
+import java.io.IOException;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Objects;
+
+import org.apache.lucene.index.LeafReaderContext;
+import org.apache.lucene.index.PostingsEnum;
+import org.apache.lucene.index.Term;
+import org.apache.lucene.index.Terms;
+import org.apache.lucene.index.TermsEnum;
+import org.apache.lucene.queries.intervals.IntervalIterator;
+import org.apache.lucene.queries.intervals.IntervalQuery;
+import org.apache.lucene.queries.intervals.IntervalsSource;
+import org.apache.lucene.search.MatchesIterator;
+import org.apache.lucene.search.Query;
+import org.apache.lucene.search.QueryVisitor;
+import org.apache.lucene.util.BytesRef;
+
+/**
+ * An IntervalsSource that returns intervals over single terms, reading token
+ * length from each term's associated payload
+ *
+ * Tokens should be indexed using {@link PayloadTokenLengthFilter} to write
+ * token lengths into payloads
+ */
+public class PayloadLengthTermIntervalsSource extends IntervalsSource {
+
+  private final BytesRef term;
+
+  /**
+   * Create a PayloadLengthTermIntervalsSource for the given term
+   */
+  public PayloadLengthTermIntervalsSource(BytesRef term) {
+this.term = term;
+  }
+
+  @Override
+  public IntervalIterator intervals(String field, LeafReaderContext ctx) 
throws IOException {
+Terms terms = ctx.reader().terms(field);
+if (terms == null)
+  return null;
+if (terms.hasPositions() == false) {
+  throw new IllegalArgumentException("Cannot create an IntervalIterator 
over field " + field + " because it has no indexed positions");
+}
+TermsEnum te = terms.iterator();
+if (te.seekExact(term) == false) {
+  return null;
+}
+return intervals(term, te);
+  }
+
+  private static int decodePayload(BytesRef payload) {
+if (payload == null) {
+  return 1;
+}
+int length = 0;
+for (int i = payload.offset; i < payload.offset + payload.length; i++) {
+  length = (length << 8) | Byte.toUnsignedInt(payload.bytes[i]);
+}
+return length;
+  }
+
+  private static IntervalIterator intervals(BytesRef term, TermsEnum te) 
throws IOException {
+PostingsEnum pe = te.postings(null, PostingsEnum.PAYLOADS);
+float cost = termPositionsCost(te);
+return new IntervalIterator() {
+
+  int length;
+
+  @Override
+  public int docID() {
+return pe.docID();
+  }
+
+  @Override
+  public int nextDoc() throws IOException {
+int doc = pe.nextDoc();
+reset();
+return doc;
+  }
+
+  @Override
+  public int advance(int target) throws IOException {
+int doc = pe.advance(target);
+reset();
+return doc;
+  }
+
+  @Override
+  public long cost() {
+return pe.cost();
+  }
+
+  int pos = -1, upto;
+
+  @Override
+  public int start() {
+return pos;
+  }
+
+  @Override
+  public int end() {
+return pos + length - 1;
+  }
+
+  @Override
+  public int gaps() {
+return 0;
+  }
+
+  @Override
+  public int nextInterval() throws IOException {
+if (upto <= 0)
+  return pos = NO_MORE_INTERVALS;
+upto--;
+pos = pe.nextPosition();
+length = decodePayload(pe.getPayload());
+return pos;
+  }
+
+  @Override
+  public float matchCost() {
+return cost;
+  }
+
+  private void reset() throws IOException {
+if (pe.docID() == NO_MORE_DOCS) {
+  upto = -1;
+  pos = NO_MORE_INTERVALS;
+}
+else {
+  upto = pe.freq();
+  pos = -1;
+}
+  }
+
+  @Override
+  public String toString() {
+return term.utf8ToString() + ":" + super.toString();
 
 Review comment:
  

[GitHub] [lucene-solr] jpountz commented on a change in pull request #769: LUCENE-8905: Better Error Handling For Illegal Arguments

[GitBox]

jpountz commented on a change in pull request #769: LUCENE-8905: Better Error 
Handling For Illegal Arguments
URL: https://github.com/apache/lucene-solr/pull/769#discussion_r302182531
 
 

 ##
 File path: lucene/core/src/java/org/apache/lucene/search/TopDocsCollector.java
 ##
 @@ -136,12 +136,9 @@ public TopDocs topDocs(int start, int howMany) {
 // pq.size() or totalHits.
 int size = topDocsSize();
 
-// Don't bother to throw an exception, just return an empty TopDocs in case
-// the parameters are invalid or out of range.
-// TODO: shouldn't we throw IAE if apps give bad params here so they dont
-// have sneaky silent bugs?
+
 if (start < 0 || start >= size || howMany <= 0) {
-  return newTopDocs(null, start);
+  throw new IllegalArgumentException("Illegal arguments when requesting 
top docs");
 
 Review comment:
   can you add the value of start and howMany to the error message and what the 
expectation is for their values?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz commented on a change in pull request #769: LUCENE-8905: Better Error Handling For Illegal Arguments

[GitBox]

jpountz commented on a change in pull request #769: LUCENE-8905: Better Error 
Handling For Illegal Arguments
URL: https://github.com/apache/lucene-solr/pull/769#discussion_r302182252
 
 

 ##
 File path: 
lucene/core/src/test/org/apache/lucene/search/TestTopDocsCollector.java
 ##
 @@ -179,6 +179,22 @@ public void testGetResultsFromStart() throws Exception {
 // get the last 5 only.
 assertEquals(5, tdc.topDocs(10).scoreDocs.length);
   }
+
+  public void testIllegalArguments() throws Exception {
+final TopDocsCollector tdc = doSearch(15);
+
+IllegalArgumentException expected = 
expectThrows(IllegalArgumentException.class, () -> {
+  tdc.topDocs(20);
+});
+
+assertEquals(expected.getMessage(), "Illegal arguments when requesting top 
docs");
 
 Review comment:
   nit: Junit assumes that the expected value is the first argument and the 
second value is what the test produced. Swapping both would produce better 
error messages.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz commented on issue #769: LUCENE-8905: Better Error Handling For Illegal Arguments

[GitBox]

jpountz commented on issue #769: LUCENE-8905: Better Error Handling For Illegal 
Arguments
URL: https://github.com/apache/lucene-solr/pull/769#issuecomment-510152347
 
 
   For instance I suspect that most of those are due to the call to `return 
TopFieldCollector.create(rewrittenSort, cappedNumHits, after, 
TOTAL_HITS_THRESHOLD);` in IndexSearcher, which we'd just need to replace with 
`return TopFieldCollector.create(rewrittenSort, cappedNumHits, after, 
Math.max(cappedNumHits, TOTAL_HITS_THRESHOLD));`?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz commented on issue #769: LUCENE-8905: Better Error Handling For Illegal Arguments

[GitBox]

jpountz commented on issue #769: LUCENE-8905: Better Error Handling For Illegal 
Arguments
URL: https://github.com/apache/lucene-solr/pull/769#issuecomment-510151784
 
 
   Can you share the stack trace of the exception? I suspect we could get rid 
of most test failures by fixing only a couple call sites.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8875) Should TopScoreDocCollector Always Populate Sentinel Values?

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882278#comment-16882278
 ] 

ASF subversion and git services commented on LUCENE-8875:
-

Commit ee79a20174528a99b1a805af5ce2212276db1630 in lucene-solr's branch 
refs/heads/master from Atri Sharma
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ee79a20 ]

LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits (#754)

This commit introduces a new collector which is optimized for
cases when the number of hits is large and/or the actual hits
collected are sparse in comparison to the number of hits
requested.

> Should TopScoreDocCollector Always Populate Sentinel Values?
> 
>
> Key: LUCENE-8875
> URL: https://issues.apache.org/jira/browse/LUCENE-8875
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Atri Sharma
>Priority: Major
>  Time Spent: 9h
>  Remaining Estimate: 0h
>
> TopScoreDocCollector always initializes HitQueue as the PQ implementation, 
> and instruct HitQueue to populate with sentinels. While this is a great 
> safety mechanism, for very large datasets where the query's selectivity is 
> high, the sentinel population can be redundant and can become a large enough 
> bottleneck in itself. Does it make sense to introduce a new parameter in 
> TopScoreDocCollector which uses a heuristic (say number of hits > 10k) and 
> does not populate sentinels?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] jpountz merged pull request #754: LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits

[GitBox]

jpountz merged pull request #754: LUCENE-8875: Introduce Optimized Collector 
For Large Number Of Hits
URL: https://github.com/apache/lucene-solr/pull/754
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13616) Possible racecondition/deadlock between collection DELETE and PrepRecovery ? (TestPolicyCloud failures)

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13616?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882245#comment-16882245
 ] 

Hoss Man commented on SOLR-13616:
-

{quote}I'm not sure we should change the waitForState logic to rethrow 
Exceptions or revert back PrepRecoveryOp to its previous version ...
{quote}
{quote}Hoss and Dat – thank you for investigating this! All usages of 
CollectionStateWatcher or LiveNodesWatcher will suffer from this problem i.e. 
the thread that runs the watcher swallows the exception ...
{quote}
Well, generally speaking there isn't any way (i can think of) for the thread 
executing a Watcher to do anything _but_ swallow any exceptions from the 
watcher – it can't propogated it back to the "caller" of registrWatcher or 
anything like that .. if the caller wanted to be informed then the Watcher it 
registered should be catching the exceptions itself.

But to Dat's point: in the specific case of {{waitForState}} – there 
ZkStateReader *is* creating it's own Watcher to wrap the input Predicate, and 
we could in fact make waitForState do something inside that Watcher that 
catches any Exception thrown by the Predicate and short circuts out of the 
{{waitForState}} call, wrapping/re-throwing the exception in the meantime.

But those seem like "broader" problems with regards to where/how the different 
callers are using the Watcher/waitForState APIs that we should probably create 
a new issue to track (for auditing all of them and clarifying the behavior in 
the javadocs) ... frankly i think in this specific jira we should be asking a 
lot more questions about the _specific_ predicate used in PrepForRecovery's 
waitForState call ... notably what exactly is the expectation here when the 
SolrCore (that prepRecovery wants to recover from) can't be found _in the local 
CoreContainer_ ... deleting the collection is just one example, are there other 
situations where the core may not be found at this point in the code? (node 
shutdown perhaps? autoscaling removing a replica) ?

what about a few lines later...
{code:java}
  if (onlyIfLeader != null && onlyIfLeader) {
if (!core.getCoreDescriptor().getCloudDescriptor().isLeader()) {
  throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, "We 
are not the leader");
}
  }
{code}
...even if the SolrCore is found, if we expect it to be the shard leader, and 
it's not (what if there has beena leader election in the meantime?) then that's 
another type of problem that will also cause the predicate to throw an 
exception that will (aparently) cause PrepRecovery to stall. what should 
PrepRecovery do here?

i suspect that in general the use of waitForState here in PrepRecoveryOp is "ok 
in concept" ... we just need to make the predicate smarter about exiting 
immeidately in these situations instead of throwing an exception that gets 
swallowed ... i'm just not sure what the right behavior for PrepRecovery *is* 
in these sitautions.

I don't suppose either of you were able to spot what's "wrong" with my test 
that it doesn't force a failure in this situation?

> Possible racecondition/deadlock between collection DELETE and PrepRecovery ? 
> (TestPolicyCloud failures)
> ---
>
> Key: SOLR-13616
> URL: https://issues.apache.org/jira/browse/SOLR-13616
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-13616.test-incomplete.patch, 
> thetaphi_Lucene-Solr-master-Linux_24358.log.txt
>
>
> Based on some recent jenkins failures in TestPolicyCloud, I suspect there is 
> a possible deadlock condition when attempting to delete a collection while 
> recovery is in progress.
> I haven't been able to identify exactly where/why/how the problem occurs, but 
> it does not appear to be a test specific problem, and seems like it could 
> potentially affect anyone unlucky enough to issue poorly timed DELETE.
> Details to follow in comments...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12368) in-place DV updates should no longer have to jump through hoops if field does not yet exist

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882226#comment-16882226
 ] 

Hoss Man commented on SOLR-12368:
-

Hey [~munendrasn] - patch functionality looks good to me, but i'm confused by 
your last comment...

bq. As part of this issue, I will commit only solr changes and raise lucene 
issue for deprecating and removing IndexWriter#getFieldNames

...but in your latest patch, IndexWriter.getFieldNames (and the underlying 
FieldInfos method) are still being removed ... shouldn't those be moved to a 
new (linked) issue/patch so that the commit for _this_ issue can be trivially 
backported?


> in-place DV updates should no longer have to jump through hoops if field does 
> not yet exist
> ---
>
> Key: SOLR-12368
> URL: https://issues.apache.org/jira/browse/SOLR-12368
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12368.patch, SOLR-12368.patch, SOLR-12368.patch
>
>
> When SOLR-5944 first added "in-place" DocValue updates to Solr, one of the 
> edge cases thta had to be dealt with was the limitation imposed by 
> IndexWriter that docValues could only be updated if they already existed - if 
> a shard did not yet have a document w/a value in the field where the update 
> was attempted, we would get an error.
> LUCENE-8316 seems to have removed this error, which i believe means we can 
> simplify & speed up some of the checks in Solr, and support this situation as 
> well, rather then falling back on full "read stored fields & reindex" atomic 
> update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-10377) Improve readability of the explain output for JSON format

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882221#comment-16882221
 ] 

Munendra S N commented on SOLR-10377:
-

Should the behavior be changed to return structured response
debug.explain.structured=true was introduced in SOLR-1915. From then to now, 
default format is changed to JSON and indent is set to on.


> Improve readability of the explain output for JSON format
> -
>
> Key: SOLR-10377
> URL: https://issues.apache.org/jira/browse/SOLR-10377
> Project: Solr
>  Issue Type: Improvement
>Reporter: Varun Thacker
>Priority: Minor
>
> Today when I ask solr for the debug query output In json with indent I get 
> this:
> {code}
> 1: " 3.545981 = sum of: 3.545981 = weight(name:dns in 0) [SchemaSimilarity], 
> result of: 3.545981 = score(doc=0,freq=1.0 = termFreq=1.0 ), product of: 
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 
> 0.5)) from: 2.0 = docFreq 24.0 = docCount 1.54 = tfNorm, computed as (freq * 
> (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 
> 1.0 = termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 
> 1.0 = fieldLength ",
> 2: " 7.4202514 = sum of: 7.4202514 = sum of: 2.7921112 = weight(name:domain 
> in 1) [SchemaSimilarity], result of: 2.7921112 = score(doc=1,freq=1.0 = 
> termFreq=1.0 ), product of: 2.3025851 = idf, computed as log(1 + (docCount - 
> docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 24.0 = docCount 
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * 
> fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 = parameter k1 
> 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 2.7921112 = 
> weight(name:name in 1) [SchemaSimilarity], result of: 2.7921112 = 
> score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 2.3025851 = idf, computed 
> as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 
> 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + 
> k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 
> = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of: 1.8360289 
> = score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 1.5141277 = idf, 
> computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 5.0 = 
> docFreq 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / 
> (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = 
> termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = 
> fieldLength "
> {code}
> When I run the same query with "wt=ruby" I get a much nicer output
> {code}
> '2'=>'
> 7.4202514 = sum of:
>   7.4202514 = sum of:
> 2.7921112 = weight(name:domain in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 2.7921112 = weight(name:name in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of:
>   1.8360289 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 1.5141277 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   5.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> ',
>   '1'=>'
> 3.545981 = sum of:
>   3.545981 = weight(name:dns in 0) [SchemaSimilarity], result of:
> 3.545981 = score(doc=0,freq=1.0 = termFreq=1.0
> ), product of:
>   2.3025851 = idf, computed as log(1 

Re: 8.1.2 bug fix release

[Steve Rowe]

I've disabled the ASF Jenkins 8.1 jobs.

--
Steve

> On Jul 10, 2019, at 9:39 AM, Đạt Cao Mạnh  wrote:
> 
> Hi Adrien,
> 
> Yeah, I am good with skipping/canceling 8.1.2 and focusing on 8.2 now.
> 
> On Wed, 10 Jul 2019 at 19:11, Adrien Grand  > wrote:
> Hi Đạt,
> 
> What do you think of focusing on 8.2 now, as Shalin suggested? Ignacio
> had suggested cutting the 8.2 branch today initially so 8.2 could be
> out pretty soon. Moreover if we want to move forward with 8.1.2, we
> will likely have to delay 8.2 at this point.
> 
> On Fri, Jul 5, 2019 at 8:42 AM Adrien Grand  > wrote:
> >
> > Agreed with Shalin, we might want to focus on 8.2 at this point.
> >
> > On Fri, Jul 5, 2019 at 8:38 AM Shalin Shekhar Mangar
> > mailto:shalinman...@gmail.com>> wrote:
> > >
> > > Thanks Dat.
> > >
> > > I don't think we should release a broken version without a fix for 
> > > SOLR-13413. A workaround for SOLR-13413 exists (forcing http1.1 for 
> > > inter-node requests) but we don't test that configuration anymore in Sole 
> > > so I am hesitant to suggest it.
> > >
> > > I think that either we agree to upgrade jetty to 9.4.19 in this point 
> > > release or we scrap it altogether and focus on 8.2.
> > >
> > > On Thu, Jul 4, 2019 at 4:54 PM Đạt Cao Mạnh  > > > wrote:
> > >>
> > >> Thanks Uwe!
> > >>
> > >> Hi guys, Ishan,
> > >> When I tryied to build the RC1 for branch_8_1. I did see this failure on 
> > >> test HttpPartitionWithTlogReplicasTest
> > >>
> > >> 215685 ERROR 
> > >> (updateExecutor-537-thread-1-processing-x:collDoRecoveryOnRestart_shard1_replica_t1
> > >>  r:core_node3 null n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart 
> > >> s:shard1) [n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart 
> > >> s:shard1 r:core_node3 x:collDoRecoveryOnRestart_shard1_replica_t1] 
> > >> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling 
> > >> SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=StdNode: 
> > >> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
> > >>  
> > >> 
> > >>  to 
> > >> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
> > >>  
> > >> 
> > >>   => java.io.IOException: java.net.ConnectException: Connection 
> > >> refused
> > >> at 
> > >> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> > >> java.io.IOException: java.net.ConnectException: Connection refused
> > >> at 
> > >> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> > >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at 
> > >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
> > >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at 
> > >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
> > >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at 
> > >> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:169) 
> > >> ~[java/:?]
> > >> at 
> > >> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:337)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> > >>  ~[java/:?]
> > >> at 
> > >> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> > >>  ~[java/:?]
> > >> at 
> > >> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> > >>  ~[metrics-core-4.0.5.jar:4.0.5]
> > >> at 
> > >> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> > >>  ~[java/:?]
> > >> at 
> > >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > >>  ~[?:1.8.0_191]
> > >> at 
> > >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > >>  ~[?:1.8.0_191]
> > >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> > >> Suppressed: java.io.IOException: java.net.ConnectException: 

[jira] [Commented] (SOLR-12368) in-place DV updates should no longer have to jump through hoops if field does not yet exist

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882192#comment-16882192
 ] 

Munendra S N commented on SOLR-12368:
-

I will commit this once 8.2 branch is cut

> in-place DV updates should no longer have to jump through hoops if field does 
> not yet exist
> ---
>
> Key: SOLR-12368
> URL: https://issues.apache.org/jira/browse/SOLR-12368
> Project: Solr
>  Issue Type: Improvement
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-12368.patch, SOLR-12368.patch, SOLR-12368.patch
>
>
> When SOLR-5944 first added "in-place" DocValue updates to Solr, one of the 
> edge cases thta had to be dealt with was the limitation imposed by 
> IndexWriter that docValues could only be updated if they already existed - if 
> a shard did not yet have a document w/a value in the field where the update 
> was attempted, we would get an error.
> LUCENE-8316 seems to have removed this error, which i believe means we can 
> simplify & speed up some of the checks in Solr, and support this situation as 
> well, rather then falling back on full "read stored fields & reindex" atomic 
> update



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8908) 2 arg "query()" does not exist for all docs, even though second arg specifies a default value

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N updated LUCENE-8908:
-
Attachment: LUCENE-8908.patch

> 2 arg "query()" does not exist for all docs, even though second arg specifies 
> a default value
> -
>
> Key: LUCENE-8908
> URL: https://issues.apache.org/jira/browse/LUCENE-8908
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Bill Bell
>Priority: Major
> Attachments: LUCENE-8908.patch, SOLR-7845.patch, SOLR-7845.patch
>
>
> The 2 arg version of the "query()" was designed so that the second argument 
> would specify the value used for any document that does not match the query 
> pecified by the first argument -- but the "exists" property of the resulting 
> ValueSource only takes into consideration wether or not the document matches 
> the query -- and ignores the use of the second argument.
> 
> The work around is to ignore the 2 arg form of the query() function, and 
> instead wrap he query function in def().
> for example:  {{def(query($something), $defaultval)}} instead of 
> {{query($something, $defaultval)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Moved] (LUCENE-8908) 2 arg "query()" does not exist for all docs, even though second arg specifies a default value

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N moved SOLR-7845 to LUCENE-8908:


Lucene Fields: New,Patch Available
  Key: LUCENE-8908  (was: SOLR-7845)
  Project: Lucene - Core  (was: Solr)

> 2 arg "query()" does not exist for all docs, even though second arg specifies 
> a default value
> -
>
> Key: LUCENE-8908
> URL: https://issues.apache.org/jira/browse/LUCENE-8908
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Bill Bell
>Priority: Major
> Attachments: SOLR-7845.patch, SOLR-7845.patch
>
>
> The 2 arg version of the "query()" was designed so that the second argument 
> would specify the value used for any document that does not match the query 
> pecified by the first argument -- but the "exists" property of the resulting 
> ValueSource only takes into consideration wether or not the document matches 
> the query -- and ignores the use of the second argument.
> 
> The work around is to ignore the 2 arg form of the query() function, and 
> instead wrap he query function in def().
> for example:  {{def(query($something), $defaultval)}} instead of 
> {{query($something, $defaultval)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7845) 2 arg "query()" does not exist for all docs, even though second arg specifies a default value

[Munendra S N (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882187#comment-16882187
 ] 

Munendra S N commented on SOLR-7845:


 [^SOLR-7845.patch] 
With additional test. I'm moving this to lucene project, as changes are in 
lucene project

> 2 arg "query()" does not exist for all docs, even though second arg specifies 
> a default value
> -
>
> Key: SOLR-7845
> URL: https://issues.apache.org/jira/browse/SOLR-7845
> Project: Solr
>  Issue Type: Bug
>Reporter: Bill Bell
>Priority: Major
> Attachments: SOLR-7845.patch, SOLR-7845.patch
>
>
> The 2 arg version of the "query()" was designed so that the second argument 
> would specify the value used for any document that does not match the query 
> pecified by the first argument -- but the "exists" property of the resulting 
> ValueSource only takes into consideration wether or not the document matches 
> the query -- and ignores the use of the second argument.
> 
> The work around is to ignore the 2 arg form of the query() function, and 
> instead wrap he query function in def().
> for example:  {{def(query($something), $defaultval)}} instead of 
> {{query($something, $defaultval)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7845) 2 arg "query()" does not exist for all docs, even though second arg specifies a default value

[Munendra S N (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Munendra S N updated SOLR-7845:
---
Attachment: SOLR-7845.patch

> 2 arg "query()" does not exist for all docs, even though second arg specifies 
> a default value
> -
>
> Key: SOLR-7845
> URL: https://issues.apache.org/jira/browse/SOLR-7845
> Project: Solr
>  Issue Type: Bug
>Reporter: Bill Bell
>Priority: Major
> Attachments: SOLR-7845.patch, SOLR-7845.patch
>
>
> The 2 arg version of the "query()" was designed so that the second argument 
> would specify the value used for any document that does not match the query 
> pecified by the first argument -- but the "exists" property of the resulting 
> ValueSource only takes into consideration wether or not the document matches 
> the query -- and ignores the use of the second argument.
> 
> The work around is to ignore the 2 arg form of the query() function, and 
> instead wrap he query function in def().
> for example:  {{def(query($something), $defaultval)}} instead of 
> {{query($something, $defaultval)}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] thomaswoeckinger commented on issue #755: SOLR-13592: Introduce EmbeddedSolrTestBase for better integration tests

[GitBox]

thomaswoeckinger commented on issue #755: SOLR-13592: Introduce 
EmbeddedSolrTestBase for better integration tests
URL: https://github.com/apache/lucene-solr/pull/755#issuecomment-510092523
 
 
   Need any support, or are you just busy at the moment?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: New feature idea - Backwards (FST) dictionary for approximate string search

[Juan Caicedo]

I didn’t integrate it directly with a Lucene index. I used the FST class
(and related utilities) to build a stand-alone spellchecker.

However, the FST for the term dictionary is modified at index time only. At
query time, we need to create a variation of the Levemshtein automata and
match it with the FST accordingly.

I need to take a closer look at the Lucene code to see how can we integrate
it with the existing modules. I’ll do that this week.

On Tue, Jul 9, 2019 at 03:18 Michael McCandless 
wrote:

> This sounds interesting!
>
> Did your patch build the FST at index time, so at search time all that was
> needed was to load the FST from the index directory?
>
> Is the FST per-segment?
>
> Mike McCandless
>
> http://blog.mikemccandless.com
>
>
> On Tue, Jul 9, 2019 at 1:43 AM Juan Caicedo 
> wrote:
>
>> Hi Michael,
>>
>> I guess that I should have added more details :-)
>>
>> The main benefit from the technique is to do approximate search in
>> large dictionaries. That is: find all the entries that are within 1 or
>> 2 edit steps from the query. It's more useful for longer (starting
>> from ~7 characters, iirc), but in general the technique can be applied
>> to queries of any length. The main requirement is that we build the
>> dictionary as an FST, using the backwards (reversed) keys of the
>> original dictionary.
>>
>> I initially used the technique to implement a stand-alone
>> spellchecker, but I think that it can also be used to optimize the
>> Lucene fuzzy queries (e.g. for the spelling/suggest module). However,
>> I'll need to look how can it be integrated with the part that creates
>> the dictionary.
>>
>> I'll take a look at the code this week and I'll try to publish it in a
>> public repository so that we can discuss about this with more concrete
>> details.
>>
>> I skimmed the paper trying to understand possible applications of the
>> technique. It sounds like efficient approximate (ie with some edits)
>> substring search is the main idea? I don't believe such a query exists
>> today in Lucene (nor any Suggester as far as I know). It sounds as if
>> this would be useful for searching within large strings, eg DNA
>> sequences or something like that, and maybe less applicable to typical
>> "full text" (ie tokenized) search where the strings being searched are
>> relatively shorter - does that sound right?
>>
>> On Sat, Jul 6, 2019 at 2:01 PM Michael Sokolov 
>> wrote:
>> >
>> > Juan, that sounds intriguing.
>> >
>> > I skimmed the paper trying to understand possible applications of the
>> > technique. It sounds like efficient approximate (ie with some edits)
>> > substring search is the main idea? I don't believe such a query exists
>> > today in Lucene (nor any Suggester as far as I know). It sounds as if
>> > this would be useful for searching within large strings, eg DNA
>> > sequences or something like that, and maybe less applicable to typical
>> > "full text" (ie tokenized) search where the strings being searched are
>> > relatively shorter - does that sound right?
>> >
>> > On Sat, Jul 6, 2019 at 12:35 PM Juan Caicedo 
>> wrote:
>> > >
>> > > Hello,
>> > >
>> > > I've been working on a project for extending LevenshteinAutomata and
>> > > I'd like to know if it would be useful to add it to Lucene.
>> > >
>> > > I've implemented the 'backwards dictionary' technique (see [1],
>> > > section 6) for speeding up approximate search. This technique allows
>> > > us to narrow down the search and, therefore, reduce the running time
>> > > (at the expense of using more memory).
>> > >
>> > > I implemented it quite some time ago using an older version of Lucene,
>> > > so I need to revisit the code. However, the implementation was
>> > > relatively simple and it didn't require major changes to the core
>> > > classes. I can share the code in a public repository and iterate on
>> > > it, while I make it compatible for new Lucene APIs, add benchmarks,
>> > > and more unit tests.
>> > >
>> > > Ideally, I'd like to contribute to Lucene, either as part of core,
>> > > suggest or a different module.
>> > >
>> > > What do you think?
>> > >
>> > > [1]
>> https://www.cis.uni-muenchen.de/download/publikationen/fastapproxsearch.pdf
>> > >
>> > > -
>> > > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > > For additional commands, e-mail: dev-h...@lucene.apache.org
>> > >
>> >
>> > -
>> > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> > For additional commands, e-mail: dev-h...@lucene.apache.org
>> >
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>>

Re: Lucene/Solr 8.2.0

[Ignacio Vera]

Thanks Tomoko for taking care of that.

On Wed, Jul 10, 2019 at 4:03 PM Đạt Cao Mạnh 
wrote:

> Hi Ignacio,
>
> 8.1.2 bugfix release will cancelled. You can go ahead with 8.2 release.
>
> Thanks!
>
> On Wed, 10 Jul 2019 at 20:38, Tomoko Uchida 
> wrote:
>
>> Hi,
>> I opened a blocker issue a while ago for release 8.2:
>> https://issues.apache.org/jira/browse/LUCENE-8907
>>
>> Sorry about that, I noticed the backwards incompatibility we have to
>> deal with today. If there are no objections, I will revert the all
>> related commits from the branch_8x and 8_2 in a few days.
>>
>> Thanks,
>> Tomoko
>>
>> 2019年7月10日(水) 22:02 Ignacio Vera :
>> >
>> > Hi,
>> >
>> > All the issues listed above has been already committed and I see no
>> blockers for release 8.2. I will cut the branch tomorrow around 10am CEST
>> and I will wait for the decision on the bug release 8.1.2 to schedule the
>> build of the first release candidate. Please let us know if this is
>> troublesome for you.
>> >
>> > Thanks,
>> >
>> > Ignacio
>> >
>> >
>> > On Tue, Jul 2, 2019 at 2:59 AM Joel Bernstein 
>> wrote:
>> >>
>> >> I've got one issue that I'd like to get in (
>> https://issues.apache.org/jira/browse/SOLR-13589), which I should have
>> wrapped up in a day or two. +1 for around July 10th.
>> >>
>> >> On Mon, Jul 1, 2019 at 5:14 PM Nicholas Knize 
>> wrote:
>> >>>
>> >>> +1 for starting the 8.2 release process. I think it would be good to
>> get the LUCENE-8632 feature into 8.2 along with the BKD improvements and
>> changes in LUCENE- and LUCENE-8896
>> >>>
>> >>> Nicholas Knize, Ph.D., GISP
>> >>> Geospatial Software Guy  |  Elasticsearch
>> >>> Apache Lucene PMC Member and Committer
>> >>> nkn...@apache.org
>> >>>
>> >>>
>> >>> On Wed, Jun 26, 2019 at 9:34 AM Ignacio Vera 
>> wrote:
>> 
>>  Hi all,
>> 
>>  8.1 has been released on May 16th and we have new features,
>> enhancements and fixes that are not released yet so I'd like to start
>> thinking in releasing Lucene/Solr 8.2.0.
>> 
>>  I can create the 8.2 branch in two weeks time (around July 10th) and
>> build the first RC by the end of that week if that works for everyone.
>> Please let me know if there are bug fixes that needs to be fixed in 8.2 and
>> might not be ready by then.
>> 
>>  Cheers,
>> 
>>  Ignacio
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
>> For additional commands, e-mail: dev-h...@lucene.apache.org
>>
>> --
> *Best regards,*
> *Cao Mạnh Đạt*
> *E-mail: caomanhdat...@gmail.com *
>

[JENKINS] Lucene-Solr-master-Windows (64bit/jdk-12.0.1) - Build # 8046 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Windows/8046/
Java: 64bit/jdk-12.0.1 -XX:-UseCompressedOops -XX:+UseConcMarkSweepGC

12 tests failed.
FAILED:  org.apache.solr.cloud.MissingSegmentRecoveryTest.testLeaderRecovery

Error Message:
IOException occurred when talking to server at: https://127.0.0.1:63466/solr

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: IOException occurred when 
talking to server at: https://127.0.0.1:63466/solr
at 
__randomizedtesting.SeedInfo.seed([DD4D14169B060849:8D188C15C227BE54]:0)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:670)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:262)
at 
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:245)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.doRequest(LBSolrClient.java:368)
at 
org.apache.solr.client.solrj.impl.LBSolrClient.request(LBSolrClient.java:296)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.sendRequest(BaseCloudSolrClient.java:1128)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.requestWithRetryOnStaleState(BaseCloudSolrClient.java:897)
at 
org.apache.solr.client.solrj.impl.BaseCloudSolrClient.request(BaseCloudSolrClient.java:829)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:228)
at 
org.apache.solr.cloud.MissingSegmentRecoveryTest.teardown(MissingSegmentRecoveryTest.java:87)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 

[jira] [Commented] (LUCENE-4312) Index format to store position length per position

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882091#comment-16882091
 ] 

Alan Woodward commented on LUCENE-4312:
---

I've opened a PR with a simple implementation of this as an IntervalsSource: 
https://github.com/apache/lucene-solr/pull/772

Note that this currently works only with index-time graphs; currently the logic 
for rewriting query-time graphs relies on the fact that the only sources with a 
minimum extent of 1 are simple terms, and this new source breaks that 
assumption.  I think we can fix that with the addition of another method to 
IntervalsSource, but I'm trying to keep things as simple as possible to begin 
with.

> Index format to store position length per position
> --
>
> Key: LUCENE-4312
> URL: https://issues.apache.org/jira/browse/LUCENE-4312
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: core/codecs
>Affects Versions: 6.0
>Reporter: Gang Luo
>Priority: Minor
>  Labels: Suggestion
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> Mike Mccandless said:TokenStreams are actually graphs.
> Indexer ignores PositionLengthAttribute.Need change the index format (and 
> Codec APIs) to store an additional int position length per position.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 8.2.0

[Đạt Cao Mạnh]

Hi Ignacio,

8.1.2 bugfix release will cancelled. You can go ahead with 8.2 release.

Thanks!

On Wed, 10 Jul 2019 at 20:38, Tomoko Uchida 
wrote:

> Hi,
> I opened a blocker issue a while ago for release 8.2:
> https://issues.apache.org/jira/browse/LUCENE-8907
>
> Sorry about that, I noticed the backwards incompatibility we have to
> deal with today. If there are no objections, I will revert the all
> related commits from the branch_8x and 8_2 in a few days.
>
> Thanks,
> Tomoko
>
> 2019年7月10日(水) 22:02 Ignacio Vera :
> >
> > Hi,
> >
> > All the issues listed above has been already committed and I see no
> blockers for release 8.2. I will cut the branch tomorrow around 10am CEST
> and I will wait for the decision on the bug release 8.1.2 to schedule the
> build of the first release candidate. Please let us know if this is
> troublesome for you.
> >
> > Thanks,
> >
> > Ignacio
> >
> >
> > On Tue, Jul 2, 2019 at 2:59 AM Joel Bernstein 
> wrote:
> >>
> >> I've got one issue that I'd like to get in (
> https://issues.apache.org/jira/browse/SOLR-13589), which I should have
> wrapped up in a day or two. +1 for around July 10th.
> >>
> >> On Mon, Jul 1, 2019 at 5:14 PM Nicholas Knize  wrote:
> >>>
> >>> +1 for starting the 8.2 release process. I think it would be good to
> get the LUCENE-8632 feature into 8.2 along with the BKD improvements and
> changes in LUCENE- and LUCENE-8896
> >>>
> >>> Nicholas Knize, Ph.D., GISP
> >>> Geospatial Software Guy  |  Elasticsearch
> >>> Apache Lucene PMC Member and Committer
> >>> nkn...@apache.org
> >>>
> >>>
> >>> On Wed, Jun 26, 2019 at 9:34 AM Ignacio Vera 
> wrote:
> 
>  Hi all,
> 
>  8.1 has been released on May 16th and we have new features,
> enhancements and fixes that are not released yet so I'd like to start
> thinking in releasing Lucene/Solr 8.2.0.
> 
>  I can create the 8.2 branch in two weeks time (around July 10th) and
> build the first RC by the end of that week if that works for everyone.
> Please let me know if there are bug fixes that needs to be fixed in 8.2 and
> might not be ready by then.
> 
>  Cheers,
> 
>  Ignacio
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
> --
*Best regards,*
*Cao Mạnh Đạt*
*E-mail: caomanhdat...@gmail.com *

[GitHub] [lucene-solr] romseygeek opened a new pull request #772: Payload-stored position length intervals

[GitBox]

romseygeek opened a new pull request #772: Payload-stored position length 
intervals
URL: https://github.com/apache/lucene-solr/pull/772
 
 
   An experimental token filter and intervals source that use payloads to encode
   token lengths


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882074#comment-16882074
 ] 

Adrien Grand commented on LUCENE-8907:
--

I have a slight preference for reverting on 8.x and have this change only in 
9.0. My worry is that the backward compatibility layer would either need to 
introduce leniency (factories without a NAME) or stronger checks (Same NAME and 
class name), and it could end up causing as much trouble as the original change.

[~thetaphi‍] What do you think?

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Assigned] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Tomoko Uchida (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tomoko Uchida reassigned LUCENE-8907:
-

Assignee: Tomoko Uchida

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Assignee: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 8.1.2 bug fix release

[Đạt Cao Mạnh]

Hi Adrien,

Yeah, I am good with skipping/canceling 8.1.2 and focusing on 8.2 now.

On Wed, 10 Jul 2019 at 19:11, Adrien Grand  wrote:

> Hi Đạt,
>
> What do you think of focusing on 8.2 now, as Shalin suggested? Ignacio
> had suggested cutting the 8.2 branch today initially so 8.2 could be
> out pretty soon. Moreover if we want to move forward with 8.1.2, we
> will likely have to delay 8.2 at this point.
>
> On Fri, Jul 5, 2019 at 8:42 AM Adrien Grand  wrote:
> >
> > Agreed with Shalin, we might want to focus on 8.2 at this point.
> >
> > On Fri, Jul 5, 2019 at 8:38 AM Shalin Shekhar Mangar
> >  wrote:
> > >
> > > Thanks Dat.
> > >
> > > I don't think we should release a broken version without a fix for
> SOLR-13413. A workaround for SOLR-13413 exists (forcing http1.1 for
> inter-node requests) but we don't test that configuration anymore in Sole
> so I am hesitant to suggest it.
> > >
> > > I think that either we agree to upgrade jetty to 9.4.19 in this point
> release or we scrap it altogether and focus on 8.2.
> > >
> > > On Thu, Jul 4, 2019 at 4:54 PM Đạt Cao Mạnh 
> wrote:
> > >>
> > >> Thanks Uwe!
> > >>
> > >> Hi guys, Ishan,
> > >> When I tryied to build the RC1 for branch_8_1. I did see this failure
> on test HttpPartitionWithTlogReplicasTest
> > >>
> > >> 215685 ERROR
> (updateExecutor-537-thread-1-processing-x:collDoRecoveryOnRestart_shard1_replica_t1
> r:core_node3 null n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart
> s:shard1) [n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart s:shard1
> r:core_node3 x:collDoRecoveryOnRestart_shard1_replica_t1]
> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling
> SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=StdNode:
> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
> to
> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
> > >>   => java.io.IOException: java.net.ConnectException:
> Connection refused
> > >> at
> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> > >> java.io.IOException: java.net.ConnectException: Connection refused
> > >> at
> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
> ~[java/:?]
> > >> at
> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
> ~[java/:?]
> > >> at
> org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:169)
> ~[java/:?]
> > >> at
> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> ~[java/:?]
> > >> at
> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> ~[java/:?]
> > >> at
> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:337)
> ~[java/:?]
> > >> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> ~[java/:?]
> > >> at
> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> ~[java/:?]
> > >> at
> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> ~[metrics-core-4.0.5.jar:4.0.5]
> > >> at
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> ~[java/:?]
> > >> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> ~[?:1.8.0_191]
> > >> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> ~[?:1.8.0_191]
> > >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> > >> Suppressed: java.io.IOException: java.net.ConnectException:
> Connection refused
> > >> at
> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
> ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> > >> at
> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
> ~[java/:?]
> > >> at
> 

Re: Lucene/Solr 8.2.0

[Tomoko Uchida]

Hi,
I opened a blocker issue a while ago for release 8.2:
https://issues.apache.org/jira/browse/LUCENE-8907

Sorry about that, I noticed the backwards incompatibility we have to
deal with today. If there are no objections, I will revert the all
related commits from the branch_8x and 8_2 in a few days.

Thanks,
Tomoko

2019年7月10日(水) 22:02 Ignacio Vera :
>
> Hi,
>
> All the issues listed above has been already committed and I see no blockers 
> for release 8.2. I will cut the branch tomorrow around 10am CEST and I will 
> wait for the decision on the bug release 8.1.2 to schedule the build of the 
> first release candidate. Please let us know if this is troublesome for you.
>
> Thanks,
>
> Ignacio
>
>
> On Tue, Jul 2, 2019 at 2:59 AM Joel Bernstein  wrote:
>>
>> I've got one issue that I'd like to get in 
>> (https://issues.apache.org/jira/browse/SOLR-13589), which I should have 
>> wrapped up in a day or two. +1 for around July 10th.
>>
>> On Mon, Jul 1, 2019 at 5:14 PM Nicholas Knize  wrote:
>>>
>>> +1 for starting the 8.2 release process. I think it would be good to get 
>>> the LUCENE-8632 feature into 8.2 along with the BKD improvements and 
>>> changes in LUCENE- and LUCENE-8896
>>>
>>> Nicholas Knize, Ph.D., GISP
>>> Geospatial Software Guy  |  Elasticsearch
>>> Apache Lucene PMC Member and Committer
>>> nkn...@apache.org
>>>
>>>
>>> On Wed, Jun 26, 2019 at 9:34 AM Ignacio Vera  wrote:

 Hi all,

 8.1 has been released on May 16th and we have new features, enhancements 
 and fixes that are not released yet so I'd like to start thinking in 
 releasing Lucene/Solr 8.2.0.

 I can create the 8.2 branch in two weeks time (around July 10th) and build 
 the first RC by the end of that week if that works for everyone. Please 
 let me know if there are bug fixes that needs to be fixed in 8.2 and might 
 not be ready by then.

 Cheers,

 Ignacio

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] iverase commented on issue #770: LUCENE-8746: Component2D topology library that works on encoded space

[GitBox]

iverase commented on issue #770: LUCENE-8746: Component2D topology library that 
works on encoded space
URL: https://github.com/apache/lucene-solr/pull/770#issuecomment-510062125
 
 
   After merging master with XYShape, it seems clear that this approach won't 
work for that encoding as it is not linear and shapes in the encoded space have 
different properties. 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13618) Solr start-up fails to discover cores outside SOLR_HOME

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882043#comment-16882043
 ] 

Shawn Heisey commented on SOLR-13618:
-

We need to be careful when deciding just how much rope we will provide for 
users that they might use to hang themselves.

Why do you want your cores in entirely different directory trees?  This might 
seem like a silly question, but we do need to assess whether there are general 
use cases, or if this feature would be rarely used and better handled in some 
other way.

The functionality you want could be achieved right now without code changes by 
putting a symlink (junction on Windows) in the solr home pointing to your 
alternate storage location.

> Solr start-up fails to discover cores outside SOLR_HOME 
> 
>
> Key: SOLR-13618
> URL: https://issues.apache.org/jira/browse/SOLR-13618
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Talvinder Matharu
>Priority: Major
>
> When new 'instanceDir' is set outside the SOLR_HOME directory then core will 
> fail to reload after restart as the server only discovers cores within 
> SOLR_HOME, looking for a 'core.properties'.  
> So what we ideally want is to check for all the “core.properties” defined 
> within all 'instanceDir' paths, which may exist outside the SOLR_HOME dir.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13617) 8.x memory issue

[Shawn Heisey (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882031#comment-16882031
 ] 

Shawn Heisey commented on SOLR-13617:
-

What is the FULL error message?  Java exceptions are typically many lines long, 
could be dozens.

This should have been brought up on the mailing list before going to Jira.  
When you started the creation process for this issue, there was a note in bold 
red letters at the top of the creation screen stating that problems should be 
discussed before opening an issue.


> 8.x memory issue
> 
>
> Key: SOLR-13617
> URL: https://issues.apache.org/jira/browse/SOLR-13617
> Project: Solr
>  Issue Type: Bug
>  Components: update
>Affects Versions: 8.0, 8.1, 8.1.1
> Environment: {color:#00}32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2.{color}
>Reporter: Scott Yeadon
>Priority: Critical
>
> {color:#00}I’m running Solr on 32-bit Ubuntu 18.04 LTS using OpenJDK 
> 10.0.2. Up until now I have had no problem with Solr (started running it 
> since 4.x), however after upgrading from 7.x to 8.x I am getting serious 
> memory issues.{color}
> {color:#00}I have a small repository of 30,000 documents currently using 
> Solr 7.1 for the search function. I attempted an upgrade to 8.1.1 and tried 
> to perform a full reindex however, it manages about 300 documents and then 
> dies from lack of memory (or so it says). I tried 8.1.0 with the same result. 
> I then tried 8.0.0 which did successfully manage a full reindex but then 
> after performing a couple of search queries died from lack of memory. I then 
> tried 7.7.2 which worked fine. I have now gone back to my original 7.1 as I 
> can’t risk 8.x in my production system.{color}
> {color:#00}I increased Xmx to 1024m (previously 512m) but that made no 
> difference, it may be some other resource than memory, but if it is, it isn’t 
> saying so, and it’s such a small repository it doesn’t seem to make sense to 
> be running out of memory, and 7.x runs fine
> {color}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Lucene/Solr 8.2.0

[Ignacio Vera]

Hi,

All the issues listed above has been already committed and I see no
blockers for release 8.2. I will cut the branch tomorrow around 10am CEST
and I will wait for the decision on the bug release 8.1.2 to schedule the
build of the first release candidate. Please let us know if this is
troublesome for you.

Thanks,

Ignacio


On Tue, Jul 2, 2019 at 2:59 AM Joel Bernstein  wrote:

> I've got one issue that I'd like to get in (
> https://issues.apache.org/jira/browse/SOLR-13589), which I should have
> wrapped up in a day or two. +1 for around July 10th.
>
> On Mon, Jul 1, 2019 at 5:14 PM Nicholas Knize  wrote:
>
>> +1 for starting the 8.2 release process. I think it would be good to get
>> the LUCENE-8632  feature
>> into 8.2 along with the BKD improvements and changes in LUCENE-
>>  and LUCENE-8896
>> 
>>
>> Nicholas Knize, Ph.D., GISP
>> Geospatial Software Guy  |  Elasticsearch
>> Apache Lucene PMC Member and Committer
>> nkn...@apache.org
>>
>>
>> On Wed, Jun 26, 2019 at 9:34 AM Ignacio Vera  wrote:
>>
>>> Hi all,
>>>
>>> 8.1 has been released on May 16th and we have new features, enhancements
>>> and fixes that are not released yet so I'd like to start thinking in
>>> releasing Lucene/Solr 8.2.0.
>>>
>>> I can create the 8.2 branch in two weeks time (around July 10th) and
>>> build the first RC by the end of that week if that works for everyone.
>>> Please let me know if there are bug fixes that needs to be fixed in 8.2 and
>>> might not be ready by then.
>>>
>>> Cheers,
>>>
>>> Ignacio
>>>
>>

[jira] [Commented] (SOLR-10377) Improve readability of the explain output for JSON format

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-10377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16882004#comment-16882004
 ] 

David Smiley commented on SOLR-10377:
-

Thanks for the reminder of that obscure parameter.  I think the main use-case 
is the user using Solr's admin screen, defaulting to json, and choosing the 
debugQuery checkbox to determine what happened with the score.  When the 
default wt (writer type / response format) was XML, the output was 
understandable.  With JSON it is not.

> Improve readability of the explain output for JSON format
> -
>
> Key: SOLR-10377
> URL: https://issues.apache.org/jira/browse/SOLR-10377
> Project: Solr
>  Issue Type: Improvement
>Reporter: Varun Thacker
>Priority: Minor
>
> Today when I ask solr for the debug query output In json with indent I get 
> this:
> {code}
> 1: " 3.545981 = sum of: 3.545981 = weight(name:dns in 0) [SchemaSimilarity], 
> result of: 3.545981 = score(doc=0,freq=1.0 = termFreq=1.0 ), product of: 
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 
> 0.5)) from: 2.0 = docFreq 24.0 = docCount 1.54 = tfNorm, computed as (freq * 
> (k1 + 1)) / (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 
> 1.0 = termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 
> 1.0 = fieldLength ",
> 2: " 7.4202514 = sum of: 7.4202514 = sum of: 2.7921112 = weight(name:domain 
> in 1) [SchemaSimilarity], result of: 2.7921112 = score(doc=1,freq=1.0 = 
> termFreq=1.0 ), product of: 2.3025851 = idf, computed as log(1 + (docCount - 
> docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 24.0 = docCount 
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - b + b * 
> fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 = parameter k1 
> 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 2.7921112 = 
> weight(name:name in 1) [SchemaSimilarity], result of: 2.7921112 = 
> score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 2.3025851 = idf, computed 
> as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 2.0 = docFreq 
> 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + 
> k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = termFreq=1.0 1.2 
> = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = fieldLength 
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of: 1.8360289 
> = score(doc=1,freq=1.0 = termFreq=1.0 ), product of: 1.5141277 = idf, 
> computed as log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5)) from: 5.0 = 
> docFreq 24.0 = docCount 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / 
> (freq + k1 * (1 - b + b * fieldLength / avgFieldLength)) from: 1.0 = 
> termFreq=1.0 1.2 = parameter k1 0.75 = parameter b 7.0 = avgFieldLength 4.0 = 
> fieldLength "
> {code}
> When I run the same query with "wt=ruby" I get a much nicer output
> {code}
> '2'=>'
> 7.4202514 = sum of:
>   7.4202514 = sum of:
> 2.7921112 = weight(name:domain in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 2.7921112 = weight(name:name in 1) [SchemaSimilarity], result of:
>   2.7921112 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 2.3025851 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   2.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> 1.8360289 = weight(name:system in 1) [SchemaSimilarity], result of:
>   1.8360289 = score(doc=1,freq=1.0 = termFreq=1.0
> ), product of:
> 1.5141277 = idf, computed as log(1 + (docCount - docFreq + 0.5) / 
> (docFreq + 0.5)) from:
>   5.0 = docFreq
>   24.0 = docCount
> 1.2125984 = tfNorm, computed as (freq * (k1 + 1)) / (freq + k1 * (1 - 
> b + b * fieldLength / avgFieldLength)) from:
>   1.0 = termFreq=1.0
>   1.2 = parameter k1
>   0.75 = parameter b
>   7.0 = avgFieldLength
>   4.0 = fieldLength
> ',
>   '1'=>'
> 3.545981 = sum of:
>   3.545981 = weight(name:dns in 0) 

[JENKINS] Lucene-Solr-SmokeRelease-8.1 - Build # 57 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-8.1/57/

No tests ran.

Build Log:
[...truncated 23880 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2570 links (2103 relative) to 3374 anchors in 253 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/solr/package/solr-8.1.2.tgz
 into 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.1/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881999#comment-16881999
 ] 

Tomoko Uchida commented on LUCENE-8907:
---

Hi,
reverting is a feasible option because there have been no changes - as far as I 
know - that depends on LUCENE-8778 other than my commits. I am sure that I can 
safely revert all my related commits so far on branch_8x (though I need to 
confirm that).
Personally I would prefer reverting instead of blocking release 8.2, which 
seems to happen soon. Maybe we can rethink the way to safely include the 
changes into 8.x again a little later. Are there any other suggestions/opinions?

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: 8.1.2 bug fix release

[Adrien Grand]

Hi Đạt,

What do you think of focusing on 8.2 now, as Shalin suggested? Ignacio
had suggested cutting the 8.2 branch today initially so 8.2 could be
out pretty soon. Moreover if we want to move forward with 8.1.2, we
will likely have to delay 8.2 at this point.

On Fri, Jul 5, 2019 at 8:42 AM Adrien Grand  wrote:
>
> Agreed with Shalin, we might want to focus on 8.2 at this point.
>
> On Fri, Jul 5, 2019 at 8:38 AM Shalin Shekhar Mangar
>  wrote:
> >
> > Thanks Dat.
> >
> > I don't think we should release a broken version without a fix for 
> > SOLR-13413. A workaround for SOLR-13413 exists (forcing http1.1 for 
> > inter-node requests) but we don't test that configuration anymore in Sole 
> > so I am hesitant to suggest it.
> >
> > I think that either we agree to upgrade jetty to 9.4.19 in this point 
> > release or we scrap it altogether and focus on 8.2.
> >
> > On Thu, Jul 4, 2019 at 4:54 PM Đạt Cao Mạnh  wrote:
> >>
> >> Thanks Uwe!
> >>
> >> Hi guys, Ishan,
> >> When I tryied to build the RC1 for branch_8_1. I did see this failure on 
> >> test HttpPartitionWithTlogReplicasTest
> >>
> >> 215685 ERROR 
> >> (updateExecutor-537-thread-1-processing-x:collDoRecoveryOnRestart_shard1_replica_t1
> >>  r:core_node3 null n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart 
> >> s:shard1) [n:127.0.0.1:55000_t_ayt%2Fs c:collDoRecoveryOnRestart s:shard1 
> >> r:core_node3 x:collDoRecoveryOnRestart_shard1_replica_t1] 
> >> o.a.s.u.ErrorReportingConcurrentUpdateSolrClient Error when calling 
> >> SolrCmdDistributor$Req: cmd=add{,id=(null)}; node=StdNode: 
> >> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/ 
> >> to 
> >> http://127.0.0.1:54997/t_ayt/s/collDoRecoveryOnRestart_shard1_replica_t2/
> >>   => java.io.IOException: java.net.ConnectException: Connection 
> >> refused
> >> at 
> >> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> >> java.io.IOException: java.net.ConnectException: Connection refused
> >> at 
> >> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
> >>  ~[java/:?]
> >> at 
> >> org.apache.solr.common.util.FastOutputStream.flushBuffer(FastOutputStream.java:209)
> >>  ~[java/:?]
> >> at org.apache.solr.common.util.JavaBinCodec.marshal(JavaBinCodec.java:169) 
> >> ~[java/:?]
> >> at 
> >> org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.marshal(JavaBinUpdateRequestCodec.java:102)
> >>  ~[java/:?]
> >> at 
> >> org.apache.solr.client.solrj.impl.BinaryRequestWriter.write(BinaryRequestWriter.java:83)
> >>  ~[java/:?]
> >> at 
> >> org.apache.solr.client.solrj.impl.Http2SolrClient.send(Http2SolrClient.java:337)
> >>  ~[java/:?]
> >> at 
> >> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.sendUpdateStream(ConcurrentUpdateHttp2SolrClient.java:231)
> >>  ~[java/:?]
> >> at 
> >> org.apache.solr.client.solrj.impl.ConcurrentUpdateHttp2SolrClient$Runner.run(ConcurrentUpdateHttp2SolrClient.java:176)
> >>  ~[java/:?]
> >> at 
> >> com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:181)
> >>  ~[metrics-core-4.0.5.jar:4.0.5]
> >> at 
> >> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
> >>  ~[java/:?]
> >> at 
> >> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>  ~[?:1.8.0_191]
> >> at 
> >> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>  ~[?:1.8.0_191]
> >> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
> >> Suppressed: java.io.IOException: java.net.ConnectException: Connection 
> >> refused
> >> at 
> >> org.eclipse.jetty.client.util.DeferredContentProvider.flush(DeferredContentProvider.java:193)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.flush(OutputStreamContentProvider.java:152)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.eclipse.jetty.client.util.OutputStreamContentProvider$DeferredOutputStream.write(OutputStreamContentProvider.java:146)
> >>  ~[jetty-client-9.4.14.v20181114.jar:9.4.14.v20181114]
> >> at 
> >> org.apache.solr.common.util.FastOutputStream.flush(FastOutputStream.java:216)
> >>  ~[java/:?]
> >> at 
> >> 

[GitHub] [lucene-solr] atris commented on issue #754: LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits

[GitBox]

atris commented on issue #754: LUCENE-8875: Introduce Optimized Collector For 
Large Number Of Hits
URL: https://github.com/apache/lucene-solr/pull/754#issuecomment-510028794
 
 
   @jpountz JFYI I ran ant beast in 3 batches of 10 times each and it ran clean:
   
   WARNING: All illegal access operations will be denied in a future release
 [beaster] Beast round 1 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/1
 [beaster] Beast round 2 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/2
 [beaster] Beast round 3 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/3
 [beaster] Beast round 4 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/4
 [beaster] Beast round 5 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/5
 [beaster] Beast round 6 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/6
 [beaster] Beast round 7 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/7
 [beaster] Beast round 8 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/8
 [beaster] Beast round 9 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/9
 [beaster] Beast round 10 results: 
/Users/atris/merge_refactor/lucene-solr/lucene/build/sandbox/test/10
 [beaster] Beasting finished Successfully.
   
   -check-totals:
   
   beast:
   
   BUILD SUCCESSFUL


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881982#comment-16881982
 ] 

Adrien Grand commented on LUCENE-8907:
--

I don't like 2, it feels too breaking to me for a minor release. I like 1 
better but then I think we should also fail if an analysis factory has a NAME 
constant that is not the same as the name that would be derived from the class 
name?

Another option could be to revert from 8.x. In any case we should add migration 
instructions to lucene/MIGRATE.txt on master.

bq. some warning messages would be helpful

We never log from Lucene since this is a library.

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13240) UTILIZENODE action results in an exception

[Christine Poerschke (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881977#comment-16881977
 ] 

Christine Poerschke commented on SOLR-13240:


bq. ... Validate source patterns validate-source-patterns failed ...

This was a case of {{"Solr test cases should extend SolrTestCase rather than 
LuceneTestCase"}} which I missed, oops. Just attached patch also removes two 
imports that are no longer needed now that the {{MoveReplicaSuggesterTest}} 
extends a base class.


> UTILIZENODE action results in an exception
> --
>
> Key: SOLR-13240
> URL: https://issues.apache.org/jira/browse/SOLR-13240
> Project: Solr
>  Issue Type: Bug
>  Components: AutoScaling
>Affects Versions: 7.6
>Reporter: Hendrik Haddorp
>Priority: Major
> Attachments: SOLR-13240.patch, SOLR-13240.patch, SOLR-13240.patch, 
> solr-solrj-7.5.0.jar
>
>
> When I invoke the UTILIZENODE action the REST call fails like this after it 
> moved a few replicas:
> {
>   "responseHeader":{
> "status":500,
> "QTime":40220},
>   "Operation utilizenode caused 
> exception:":"java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
>  Comparison method violates its general contract!",
>   "exception":{
> "msg":"Comparison method violates its general contract!",
> "rspCode":-1},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"Comparison method violates its general contract!",
> "trace":"org.apache.solr.common.SolrException: Comparison method violates 
> its general contract!\n\tat 
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
>  
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
>  
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
>  org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
>  
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
>  
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
>  
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
>  
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
>  
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
>  org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat 
> 

[jira] [Updated] (SOLR-13240) UTILIZENODE action results in an exception

[Christine Poerschke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Christine Poerschke updated SOLR-13240:
---
Attachment: SOLR-13240.patch

> UTILIZENODE action results in an exception
> --
>
> Key: SOLR-13240
> URL: https://issues.apache.org/jira/browse/SOLR-13240
> Project: Solr
>  Issue Type: Bug
>  Components: AutoScaling
>Affects Versions: 7.6
>Reporter: Hendrik Haddorp
>Priority: Major
> Attachments: SOLR-13240.patch, SOLR-13240.patch, SOLR-13240.patch, 
> solr-solrj-7.5.0.jar
>
>
> When I invoke the UTILIZENODE action the REST call fails like this after it 
> moved a few replicas:
> {
>   "responseHeader":{
> "status":500,
> "QTime":40220},
>   "Operation utilizenode caused 
> exception:":"java.lang.IllegalArgumentException:java.lang.IllegalArgumentException:
>  Comparison method violates its general contract!",
>   "exception":{
> "msg":"Comparison method violates its general contract!",
> "rspCode":-1},
>   "error":{
> "metadata":[
>   "error-class","org.apache.solr.common.SolrException",
>   "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"Comparison method violates its general contract!",
> "trace":"org.apache.solr.common.SolrException: Comparison method violates 
> its general contract!\n\tat 
> org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:274)\n\tat
>  
> org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:246)\n\tat
>  
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)\n\tat
>  
> org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)\n\tat 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)\n\tat
>  org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)\n\tat 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:377)\n\tat
>  
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:323)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
>  
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
>  
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
>  
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)\n\tat
>  
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)\n\tat
>  
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
>  
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:219)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
>  org.eclipse.jetty.server.Server.handle(Server.java:531)\n\tat 
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)\n\tat 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)\n\tat
>  
> org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)\n\tat
>  org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)\n\tat 
> org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)\n\tat 
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
>  
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
>  
> org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)\n\tat
>  

[jira] [Commented] (LUCENE-8311) Leverage impacts for phrase queries

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881967#comment-16881967
 ] 

Adrien Grand commented on LUCENE-8311:
--

This made exact phrase queries 3x faster in the nightly benchmarks 
http://people.apache.org/~mikemccand/lucenebench/Phrase.html and term queries 
about 10% slower http://people.apache.org/~mikemccand/lucenebench/Term.html.

> Leverage impacts for phrase queries
> ---
>
> Key: LUCENE-8311
> URL: https://issues.apache.org/jira/browse/LUCENE-8311
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Attachments: LUCENE-8311.patch
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Now that we expose raw impacts, we could leverage them for phrase queries.
> For instance for exact phrases, we could take the minimum term frequency for 
> each unique norm value in order to get upper bounds of the score for the 
> phrase.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-13618) Solr start-up fails to discover cores outside SOLR_HOME

[Talvinder Matharu (JIRA)]

Talvinder Matharu created SOLR-13618:


 Summary: Solr start-up fails to discover cores outside SOLR_HOME 
 Key: SOLR-13618
 URL: https://issues.apache.org/jira/browse/SOLR-13618
 Project: Solr
  Issue Type: Bug
  Security Level: Public (Default Security Level. Issues are Public)
Reporter: Talvinder Matharu


When new 'instanceDir' is set outside the SOLR_HOME directory then core will 
fail to reload after restart as the server only discovers cores within 
SOLR_HOME, looking for a 'core.properties'.  

So what we ideally want is to check for all the “core.properties” defined 
within all 'instanceDir' paths, which may exist outside the SOLR_HOME dir.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8907) Provide backward compatibility for loading analysis factories

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881944#comment-16881944
 ] 

Tomoko Uchida commented on LUCENE-8907:
---

There may be two options:

1. Completely emulate the factory loading method of Lucene 8.1. (i.e., copy the 
old logic and use it if the "NAME" field does not exist in a factory.)
2. Ignore factories which have no proper "NAME" field. (We have to note the 
behaviour somewhere.)

Either way, some warning messages would be helpful to let users know that they 
have to change their code by Lucene 9.0 (can we show such "warnings"?).

> Provide backward compatibility for loading analysis factories
> -
>
> Key: LUCENE-8907
> URL: https://issues.apache.org/jira/browse/LUCENE-8907
> Project: Lucene - Core
>  Issue Type: Task
>  Components: modules/analysis
>Reporter: Tomoko Uchida
>Priority: Blocker
>
> The changes in LUCENE-8778 have breaking changes in the analysis factory 
> interface and  custom factories implemented by users / 3rd parties will be 
> affected. We need to keep some backwards compatibility during 8.x.
> Please see the discussion in SOLR-13593 for details.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] [lucene-solr] atris commented on issue #754: LUCENE-8875: Introduce Optimized Collector For Large Number Of Hits

[GitBox]

atris commented on issue #754: LUCENE-8875: Introduce Optimized Collector For 
Large Number Of Hits
URL: https://github.com/apache/lucene-solr/pull/754#issuecomment-510007530
 
 
   > @atris Can you give some result comparison or other evidence?
   
   Please see the blog post listed above.
   
   In addition, the targeted use case for this collector makes the benefits 
obvious: I ran a TermQuery on a 5 TB dataset manufactured from Elasticsearch's 
nyc_taxis dataset. A string field was modified to have a specific value only 
for 50,000 documents, and number of hits requested was 25,00,000 by each of 30 
concurrent requests (the same test mentioned in Toke's blogpost). The space 
savings were in order of multiple GBs, as expected.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5837) Add missing equals implementation for SolrDocument, SolrInputDocument and SolrInputField.

[jia yang (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881908#comment-16881908
 ] 

jia yang commented on SOLR-5837:


 
[https://svn.apache.org/viewvc/lucene/dev/branches/branch_4x/solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java?view=markup=1577500]
    

I find  SolrTestCaseJ4.java with a bug . assertSolrDocumentEquals  method  1873 
row .  if solr result  Is disordered，then 

!key1.equals(key2) must be true . 

so i repair:

String key1 = iter1.next();


Object val1 = solrDocument1.getFieldValues(key1);

Object val2 = solrDocument2.getFieldValues(key1);

if (!val1.equals(val2)) {
 return false;
}

 

> Add missing equals implementation for SolrDocument, SolrInputDocument and 
> SolrInputField.
> -
>
> Key: SOLR-5837
> URL: https://issues.apache.org/jira/browse/SOLR-5837
> Project: Solr
>  Issue Type: Improvement
>Reporter: Varun Thacker
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-5837.patch, SOLR-5837.patch
>
>
> While working on SOLR-5265 I tried comparing objects of SolrDocument, 
> SolrInputDocument and SolrInputField. These classes did not Override the 
> equals implementation. 
> The issue will Override equals and hashCode methods to the 3 classes.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13593) Allow to specify analyzer components by their SPI names in schema definition

[Tomoko Uchida (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881892#comment-16881892
 ] 

Tomoko Uchida commented on SOLR-13593:
--

I opened an blocker issue: LUCENE-8907.

[~thetaphi]: Currently I am not sure about how to properly deal with the 
backwards compatibility. Can you please give some directions?

> Allow to specify analyzer components by their SPI names in schema definition
> 
>
> Key: SOLR-13593
> URL: https://issues.apache.org/jira/browse/SOLR-13593
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: Schema and Analysis
>Reporter: Tomoko Uchida
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Now each analysis factory has explicitely documented SPI name which is stored 
> in the static "NAME" field (LUCENE-8778).
>  Solr uses factories' simple class name in schema definition (like 
> class="solr.WhitespaceTokenizerFactory"), but we should be able to also use 
> more concise SPI names (like name="whitespace").
> e.g.:
> {code:xml}
> 
>   
> 
>  />
> 
>   
> 
> {code}
> would be
> {code:xml}
> 
>   
> 
> 
> 
>   
> 
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org