Re: Solr Size Limitation upto 32 kb limitation

[Kranthi Kumar K]

Hi Team,


Can we have any updates on the below issue? We are awaiting your reply.


Thanks,

Kranthi kumar.K


From: Kranthi Kumar K
Sent: Friday, January 4, 2019 5:01:38 PM
To: dev@lucene.apache.org
Cc: Ananda Babu medida; Srinivasa Reddy Karri
Subject: Solr Size Limitation upto 32 kb limitation


Hi team,



We are currently using Solr 4.2.1 version in our project and everything is 
going well. But recently, we are facing an issue with Solr Data Import. It is 
not importing the files with size greater than 32766 bytes (i.e, 32 kb) and 
showing 2 exceptions:



  1.  java.lang.illegalargumentexception
  2.  org.apache.lucene.util.bytesref hash$maxbyteslengthexceededexception



Please find the attached screenshot for reference.



We have searched for solutions in many forums and didn’t find the exact 
solution for this issue. Interestingly, we found in the article, by changing 
the type of the ‘field’ from sting to  ‘text_general’ might solve the issue. 
Please have a look in the below forum:



https://stackoverflow.com/questions/29445323/adding-a-document-to-the-index-in-solr-document-contains-at-least-one-immense-t



Schema.xml:

Changed from:

‘’



Changed to:

‘’



We have tried it but still it is not importing the files > 32 KB or 32766 bytes.



Could you please let us know the solution to fix this issue? We’ll be awaiting 
your reply.

[jira] [Resolved] (SOLR-12806) when strict=false is specified, prioritize node allocation using non strict rules

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-12806.
---
   Resolution: Fixed
Fix Version/s: 7.6

> when strict=false is specified, prioritize node allocation using non strict 
> rules
> -
>
> Key: SOLR-12806
> URL: https://issues.apache.org/jira/browse/SOLR-12806
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Fix For: 7.6
>
> Attachments: SOLR-12806.patch
>
>
> if a rule is specified as
> for instance if a policy rule as follows exists
> {code:java}
> {"replica" : "#ALL", "freedisk" : "<500", "strict" : false}
> {code}
>  
> If no no nodes have {{freedisk}} more than 500 GB, Solr ignores this rule 
> completely and assign nodes. Ideally it should still prefer a node with 
> {{freedisk}} of 450GB compared to a node that has {{freedisk}} of 400GB



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12023) Autoscaling policy engine shuffles replicas needlessly and can also suggest nonexistent replicas to be moved

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-12023.
---
Resolution: Fixed

> Autoscaling policy engine shuffles replicas needlessly and can also suggest 
> nonexistent replicas to be moved
> 
>
> Key: SOLR-12023
> URL: https://issues.apache.org/jira/browse/SOLR-12023
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>Priority: Major
> Fix For: 7.6, 8.0
>
> Attachments: SOLR-11066-failing.patch, SOLR-12023.patch
>
>
> A test that I wrote in SOLR-11066 found the following problem:
> Cluster: 2 nodes
> Collection: 1 shard, 3 replicas, maxShardsPerNode=5
> No autoscaling policy or preference applied
> When the trigger runs, the computed plan needlessly shuffles all three 
> replicas and then proceeds to return suggestions with only numbers as core 
> names. These cores do not exist. I found that these numbers are generated 
> internally by the framework as placeholders for moved cores for further 
> calculations. They should never ever be suggested to the users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12977) Autoscaling policy initialisation tries to fetch metrics from dead nodes

[Noble Paul (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Noble Paul resolved SOLR-12977.
---
Resolution: Fixed

> Autoscaling policy initialisation tries to fetch metrics from dead nodes
> 
>
> Key: SOLR-12977
> URL: https://issues.apache.org/jira/browse/SOLR-12977
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>Priority: Minor
> Fix For: 7.6, 8.0
>
>
> Autoscaling policy initialisation tries to fetch metrics for each node during 
> construction. However, it does not skip the known dead nodes causing a 
> timeout to be logged. We should skip such requests entirely.
> {code}
> 20579 WARN  (AutoscalingActionExecutor-37-thread-1) [] 
> o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node 
> 127.0.0.1:63255_solr
> org.apache.solr.client.solrj.SolrServerException: Server refused connection 
> at: http://127.0.0.1:63255/solr
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:650)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) 
> ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:186)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getReplicaInfo(SolrClientNodeStateProvider.java:169)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.cloud.autoscaling.Row.(Row.java:60) 
> [java/:?]
>   at 
> org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:181)
>  [java/:?]
>   at 
> org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:114)
>  [java/:?]
>   at 
> org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:308)
>  [java/:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[mosh (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744691#comment-16744691
 ] 

mosh commented on SOLR-5211:


{quote}What if we delete the complete block and create the block again and add 
again via UpdateChainProcessorFactory. In that way, will update only those 
fields/docs what we want to and put the block back in place
{quote}
Have you had a look at SOLR-12638?
 It allows deleting and adding child docs as an atomic update.
{quote}That's one workaround I am planning to try (on 7.5) – delete the parent 
& children, then re-add. Unfortunately, I believe this will take two 
calls.{quote}
I am not sure whether 
[SOLR-12638|https://issues.apache.org/jira/browse/SOLR-12638] will make it to 
Solr 7.

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 2687 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/2687/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-SmokeRelease-master/1235/consoleText

[repro] Revision: d970375cd2f2357b6a8da5ac67ef994f8d43

[repro] Ant options: -DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9
[repro] Repro line:  ant test  -Dtestcase=AutoscalingHistoryHandlerTest 
-Dtests.seed=7C23CC22AFEE9DE2 -Dtests.multiplier=2 -Dtests.locale=da 
-Dtests.timezone=Pacific/Saipan -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
e35adf6824b091cdf39984a7e426c2e886f56552
[repro] git fetch
[repro] git checkout d970375cd2f2357b6a8da5ac67ef994f8d43

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   AutoscalingHistoryHandlerTest
[repro] ant compile-test

[...truncated 3592 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.AutoscalingHistoryHandlerTest" -Dtests.showOutput=onerror 
-DsmokeTestRelease.java9=/home/jenkins/tools/java/latest1.9 
-Dtests.seed=7C23CC22AFEE9DE2 -Dtests.multiplier=2 -Dtests.locale=da 
-Dtests.timezone=Pacific/Saipan -Dtests.asserts=true -Dtests.file.encoding=UTF-8

[...truncated 3736 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   1/5 failed: 
org.apache.solr.handler.admin.AutoscalingHistoryHandlerTest
[repro] git checkout e35adf6824b091cdf39984a7e426c2e886f56552

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 6 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-BadApples-NightlyTests-7.x - Build # 44 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-BadApples-NightlyTests-7.x/44/

3 tests failed.
FAILED:  org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test

Error Message:
The Monkey ran for over 60 seconds and no jetties were stopped - this is worth 
investigating!

Stack Trace:
java.lang.AssertionError: The Monkey ran for over 60 seconds and no jetties 
were stopped - this is worth investigating!
at 
__randomizedtesting.SeedInfo.seed([C3826F1E4C04A535:4BD650C4E2F8C8CD]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.apache.solr.cloud.ChaosMonkey.stopTheMonkey(ChaosMonkey.java:541)
at 
org.apache.solr.cloud.ChaosMonkeyNothingIsSafeTest.test(ChaosMonkeyNothingIsSafeTest.java:207)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsFixedStatement.callStatement(BaseDistributedSearchTestCase.java:1075)
at 
org.apache.solr.BaseDistributedSearchTestCase$ShardsRepeatRule$ShardsStatement.evaluate(BaseDistributedSearchTestCase.java:1047)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Issue Comment Deleted] (SOLR-5211) updating parent as childless makes old children orphans

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated SOLR-5211:
---
Comment: was deleted

(was: Yes, that will take 2 calls, I have implemented this with 7.2.1 and 6.6, 
Patch is attached in this  https://issues.apache.org/jira/browse/SOLR-13062.)

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12424) Search trigger does not accept handler

[Cassandra Targett (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett resolved SOLR-12424.
--
   Resolution: Fixed
Fix Version/s: 7.4

The documentation was updated for 7.4, so I think this was just a docs mistake.

> Search trigger does not accept handler
> --
>
> Key: SOLR-12424
> URL: https://issues.apache.org/jira/browse/SOLR-12424
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Varun Thacker
>Priority: Major
> Fix For: 7.4
>
>
> This ref guide page ( 
> [http://lucene.apache.org/solr/guide/7_3/solrcloud-autoscaling-triggers.html#search-rate-trigger]
>  ) provides an example JSON for a search trigger.
> If I post that on master I get the following error
> {code:java}
> ERROR - 2018-05-30 02:29:54.837; [   ] 
> org.apache.solr.handler.RequestHandlerBase; 
> org.apache.solr.api.ApiBag$ExceptionWithErrObject: Error in command payload, 
> errors: [{set-trigger={name=search_rate_trigger, event=searchRate, 
> collection=test, handler=/select, rate=100.0, waitFor=1m, enabled=true, 
> actions=[{name=compute_plan, class=solr.ComputePlanAction}, 
> {name=execute_plan, class=solr.ExecutePlanAction}]}, errorMessages=[Error 
> validating trigger config search_rate_trigger: 
> TriggerValidationException{name=search_rate_trigger, 
> details='{handler=unknown property}'}]}], 
>  at 
> org.apache.solr.cloud.autoscaling.AutoScalingHandler.processOps(AutoScalingHandler.java:210)
>  at 
> org.apache.solr.cloud.autoscaling.AutoScalingHandler.handleRequestBody(AutoScalingHandler.java:148)
>  at 
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:734)
>  at 
> org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:715)
>  at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:378)
>  at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:324)
>  at 
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1634)
>  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
>  at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> ...{code}
> From the JSON payload if I remove the "handler" key it works. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12815) Implement maxOps limit for IndexSizeTrigger

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744644#comment-16744644
 ] 

Cassandra Targett commented on SOLR-12815:
--

[~ab], this appears in CHANGES.txt for 7.6, can it be resolved?

> Implement maxOps limit for IndexSizeTrigger
> ---
>
> Key: SOLR-12815
> URL: https://issues.apache.org/jira/browse/SOLR-12815
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Andrzej Bialecki 
>Assignee: Andrzej Bialecki 
>Priority: Major
>
> {{IndexSizeTrigger}} can currently generate unlimited number of requested 
> operations (such as split shard). Combined with the avalanche effect in 
> SOLR-12730 this may cause a massive spike in the cluster load, and some of 
> these operations may fail.
> It's probably better to put an arbitrary limit on the maximum number of 
> generated ops requested in one trigger event. This will delay some of the 
> needed changes (thus leading to thresholds being exceeded to a larger degree) 
> but it will increase the likelihood that all operations succeed.
> A similar limit already exists in {{SearchRateTrigger}}.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8642) RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744639#comment-16744639
 ] 

Mike Sokolov commented on LUCENE-8642:
--

I feel like there is a value to a partial solution here, throwing an exception 
for unhandled types. I even think it has value when the estimates are not 
perfect (like for HashMap, where our accounting for internal Entry objects is 
pretty crude). It may not be perfect, but sometimes you really do want a tool 
for this, and there is nothing else quite like RamUsageEstimator :). I'm more 
thinking of cases where you want to optimize heap usage, rather than unit tests 
_per se_. Even for unit tests, it can be useful to be able to assert that RAM 
usage is under some bound to guard against future scaling mess-ups, and getting 
a precise measurement may not always be critical.

> RamUsageTester.sizeOf ignores arrays and collections if --illegal-access=deny
> -
>
> Key: LUCENE-8642
> URL: https://issues.apache.org/jira/browse/LUCENE-8642
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>
> We have a silent fall-through on catch clause there if a field is 
> inaccessible. This causes all collections to not be recursively descended 
> into.
> We could hack it so that collection objects are counted "manually". Or we 
> could propagate illegal access exception from ram tester, detect which tests/ 
> objects do try to measure collection objects and either remove them (if it's 
> not possible to measure accurately) or change them (if it is possible to 
> estimate the size in a different way).
> Looking for feedback on this one.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744621#comment-16744621
 ] 

Mike Sokolov commented on LUCENE-8635:
--

I used the wikimedia2m data set for the second set of tests (the first test was 
on a tiny index - 10k docs) -- at least I think I did! I am kind of new to the 
benchmarking game. I ran the becnhmarks with {{python src/python/localrun.py 
-source wikimedium2m}}, and I can see that the index dir is 861M.





> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: [jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Michael Sokolov]

I used the wikimedia2m data set for the second set of tests (the first test
was on a tiny index - 10k docs) -- at least I think I did! I am kind of new
to the benchmarking game. I ran the becnhmarks with python
src/python/localrun.py -source wikimedium2m, and I can see that the index
dir is 861M.


On Wed, Jan 16, 2019 at 7:18 PM Michael McCandless (JIRA) 
wrote:

>
> [
> https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744538#comment-16744538
> ]
>
> Michael McCandless commented on LUCENE-8635:
> 
>
> Thanks [~sokolov] – those numbers look quite a bit better!  Though, your
> QPSs are kinda high overall – how many Wikipedia docs were in your index?
>
> I do wonder if we simply reversed the FST's byte[] when we create it, what
> impact that'd have on lookup performance.  Hmm even if we did that, we'd
> still have to {{readBytes}} one byte at a time since {{RandomAccessInput}}
> does not have a {{readBytes}} method?  But ... maybe {{IndexInput}} would
> give good performance in that case?  We should probably pursue that
> separately though...
>
> > Lazy loading Lucene FST offheap using mmap
> > --
> >
> > Key: LUCENE-8635
> > URL: https://issues.apache.org/jira/browse/LUCENE-8635
> > Project: Lucene - Core
> >  Issue Type: New Feature
> >  Components: core/FSTs
> > Environment: I used below setup for es_rally tests:
> > single node i3.xlarge running ES 6.5
> > es_rally was running on another i3.xlarge instance
> >Reporter: Ankit Jain
> >Priority: Major
> > Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
> >
> >
> > Currently, FST loads all the terms into heap memory during index open.
> This causes frequent JVM OOM issues if the term size gets big. A better way
> of doing this will be to lazily load FST using mmap. That ensures only the
> required terms get loaded into memory.
> >
> > Lucene can expose API for providing list of fields to load terms
> offheap. I'm planning to take following approach for this:
> >  # Add a boolean property fstOffHeap in FieldInfo
> >  # Pass list of offheap fields to lucene during index open (ALL can be
> special keyword for loading ALL fields offheap)
> >  # Initialize the fstOffHeap property during lucene index open
> >  # FieldReader invokes default FST constructor or OffHeap constructor
> based on fstOffHeap field
> >
> > I created a patch (that loads all fields offheap), did some benchmarks
> using es_rally and results look good.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v7.6.3#76005)
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
> For additional commands, e-mail: dev-h...@lucene.apache.org
>
>

[GitHub] MighTguY commented on issue #537: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY commented on issue #537: LUCENE-8640: added changes for the validation 
of valid dateString
URL: https://github.com/apache/lucene-solr/pull/537#issuecomment-455009248
 
 
   Raised another PR https://github.com/apache/lucene-solr/pull/538


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] MighTguY commented on issue #537: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY commented on issue #537: LUCENE-8640: added changes for the validation 
of valid dateString
URL: https://github.com/apache/lucene-solr/pull/537#issuecomment-455009176
 
 
   Closing this request, Due to my forked repo issues.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] MighTguY closed pull request #537: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY closed pull request #537: LUCENE-8640: added changes for the 
validation of valid dateString
URL: https://github.com/apache/lucene-solr/pull/537
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] MighTguY opened a new pull request #538: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY opened a new pull request #538: LUCENE-8640: added changes for the 
validation of valid dateString
URL: https://github.com/apache/lucene-solr/pull/538
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] MighTguY closed pull request #538: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY closed pull request #538: LUCENE-8640: added changes for the 
validation of valid dateString
URL: https://github.com/apache/lucene-solr/pull/538
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] MighTguY opened a new pull request #538: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY opened a new pull request #538: LUCENE-8640: added changes for the 
validation of valid dateString
URL: https://github.com/apache/lucene-solr/pull/538
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13007) Use javabin instead of JSON to send messages to overseer

[Noble Paul (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744575#comment-16744575
 ] 

Noble Paul commented on SOLR-13007:
---

The fix itself is pretty easy. I didn't do it because it can cause backcompat 
issues. That's why I committed SOLR-13006 first. If all nodes are already 
upgraded to solr 7.6 or newer then we don't have to worry about backcompat 
problem.

> Use javabin instead of JSON to send messages to overseer
> 
>
> Key: SOLR-13007
> URL: https://issues.apache.org/jira/browse/SOLR-13007
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Priority: Major
>
> The messages themselves are ephemeral and the readability is not a big issue. 
> Using javabin can:
> * reduce the payload size
> * make processing faster



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12467) allow to change the autoscaling configuration via SolrJ

[Cassandra Targett (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett updated SOLR-12467:
-
Component/s: AutoScaling

> allow to change the autoscaling configuration via SolrJ
> ---
>
> Key: SOLR-12467
> URL: https://issues.apache.org/jira/browse/SOLR-12467
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling, SolrJ
>Affects Versions: 7.3.1
>Reporter: Hendrik Haddorp
>Priority: Minor
>
> Using SolrJ's CloudSolrClient it is possible to read the autoscaling 
> configuration:
> cloudSolrClient.getZkStateReader().getAutoScalingConfig()
> There is however no way to update it. One can only read out the list of life 
> nodes and then do a call to Solr using the LBHttpSolrClient for example. 
> Given that the config is stored in ZooKeeper and thus could be written 
> directly and even when no Solr instance is running this is not optimal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12868) Request forwarding for v2 API is broken

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744568#comment-16744568
 ] 

Cassandra Targett commented on SOLR-12868:
--

[~noble.paul], here's *another* issue that appears in CHANGES.txt for 7.6 but 
isn't marked as resolved. Can it be resolved?

It seems to happen frequently that we find issues that should have been 
resolved months ago. Any way you could modify the way you work with issues so 
these are resolved closer to the time of your commits, like the rest of us do?

> Request forwarding for v2 API is broken
> ---
>
> Key: SOLR-12868
> URL: https://issues.apache.org/jira/browse/SOLR-12868
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: SolrCloud, v2 API
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>Priority: Major
> Fix For: 7.6, 8.0
>
>
> I was working with Noble Paul to investigate test failures seen in SOLR-12806 
> where we found this issue. Due to a bug, replicas of a collection weren't 
> spread evenly so there were some nodes which did not have any replicas at 
> all. In such cases, when a v2 API call hits an empty node, it is not 
> forwarded to the right path on the remote node causing test failures.
> e.g. a call to {{/c/collection/_introspect}} is forwarded as 
> {{http://127.0.0.1:63326/solr/collection1/_introspect?method=POST=javabin=2=}}
>  and {{/c/collection1/abccdef}} is forwarded as 
> {{http://127.0.0.1:63326/solr/collection1/abccdef}}
> In summary, a remote query for v2 API from an empty node is converted to a v1 
> style call which may not be a valid path. We should forward v2 API calls 
> as-is without changing the paths.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-Tests-8.x - Build # 7 - Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-Tests-8.x/7/

2 tests failed.
FAILED:  
org.apache.solr.cloud.DeleteReplicaTest.deleteReplicaFromClusterStateLegacy

Error Message:
Timeout waiting for replica get deleted Timeout waiting to see state for 
collection=deleteFromClusterState_true 
:DocCollection(deleteFromClusterState_true//collections/deleteFromClusterState_true/state.json/7)={
   "pullReplicas":"0",   "replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node4":{   
"core":"deleteFromClusterState_true_shard1_replica_n1",   
"base_url":"http://127.0.0.1:39333/solr;,   
"node_name":"127.0.0.1:39333_solr",   "state":"active",   
"type":"NRT"}, "core_node6":{   
"core":"deleteFromClusterState_true_shard1_replica_n3",   
"base_url":"http://127.0.0.1:44940/solr;,   
"node_name":"127.0.0.1:44940_solr",   "state":"active",   
"type":"NRT",   "leader":"true"}, "core_node5":{   
"core":"deleteFromClusterState_true_shard1_replica_n2",   
"base_url":"http://127.0.0.1:45668/solr;,   
"node_name":"127.0.0.1:45668_solr",   "state":"active",   
"type":"NRT",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",  
 "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"} Live 
Nodes: [127.0.0.1:39333_solr, 127.0.0.1:41208_solr, 127.0.0.1:44940_solr, 
127.0.0.1:45668_solr] Last available state: 
DocCollection(deleteFromClusterState_true//collections/deleteFromClusterState_true/state.json/7)={
   "pullReplicas":"0",   "replicationFactor":"3",   "shards":{"shard1":{   
"range":"8000-7fff",   "state":"active",   "replicas":{ 
"core_node4":{   
"core":"deleteFromClusterState_true_shard1_replica_n1",   
"base_url":"http://127.0.0.1:39333/solr;,   
"node_name":"127.0.0.1:39333_solr",   "state":"active",   
"type":"NRT"}, "core_node6":{   
"core":"deleteFromClusterState_true_shard1_replica_n3",   
"base_url":"http://127.0.0.1:44940/solr;,   
"node_name":"127.0.0.1:44940_solr",   "state":"active",   
"type":"NRT",   "leader":"true"}, "core_node5":{   
"core":"deleteFromClusterState_true_shard1_replica_n2",   
"base_url":"http://127.0.0.1:45668/solr;,   
"node_name":"127.0.0.1:45668_solr",   "state":"active",   
"type":"NRT",   "router":{"name":"compositeId"},   "maxShardsPerNode":"1",  
 "autoAddReplicas":"false",   "nrtReplicas":"3",   "tlogReplicas":"0"}

Stack Trace:
java.lang.AssertionError: Timeout waiting for replica get deleted
Timeout waiting to see state for collection=deleteFromClusterState_true 
:DocCollection(deleteFromClusterState_true//collections/deleteFromClusterState_true/state.json/7)={
  "pullReplicas":"0",
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node4":{
  "core":"deleteFromClusterState_true_shard1_replica_n1",
  "base_url":"http://127.0.0.1:39333/solr;,
  "node_name":"127.0.0.1:39333_solr",
  "state":"active",
  "type":"NRT"},
"core_node6":{
  "core":"deleteFromClusterState_true_shard1_replica_n3",
  "base_url":"http://127.0.0.1:44940/solr;,
  "node_name":"127.0.0.1:44940_solr",
  "state":"active",
  "type":"NRT",
  "leader":"true"},
"core_node5":{
  "core":"deleteFromClusterState_true_shard1_replica_n2",
  "base_url":"http://127.0.0.1:45668/solr;,
  "node_name":"127.0.0.1:45668_solr",
  "state":"active",
  "type":"NRT",
  "router":{"name":"compositeId"},
  "maxShardsPerNode":"1",
  "autoAddReplicas":"false",
  "nrtReplicas":"3",
  "tlogReplicas":"0"}
Live Nodes: [127.0.0.1:39333_solr, 127.0.0.1:41208_solr, 127.0.0.1:44940_solr, 
127.0.0.1:45668_solr]
Last available state: 
DocCollection(deleteFromClusterState_true//collections/deleteFromClusterState_true/state.json/7)={
  "pullReplicas":"0",
  "replicationFactor":"3",
  "shards":{"shard1":{
  "range":"8000-7fff",
  "state":"active",
  "replicas":{
"core_node4":{
  "core":"deleteFromClusterState_true_shard1_replica_n1",
  "base_url":"http://127.0.0.1:39333/solr;,
  "node_name":"127.0.0.1:39333_solr",
  "state":"active",
  "type":"NRT"},
"core_node6":{
  "core":"deleteFromClusterState_true_shard1_replica_n3",
  "base_url":"http://127.0.0.1:44940/solr;,
  "node_name":"127.0.0.1:44940_solr",
  "state":"active",
  "type":"NRT",
  "leader":"true"},
"core_node5":{
  "core":"deleteFromClusterState_true_shard1_replica_n2",
  

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Michael McCandless (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744538#comment-16744538
 ] 

Michael McCandless commented on LUCENE-8635:


Thanks [~sokolov] – those numbers look quite a bit better!  Though, your QPSs 
are kinda high overall – how many Wikipedia docs were in your index?

I do wonder if we simply reversed the FST's byte[] when we create it, what 
impact that'd have on lookup performance.  Hmm even if we did that, we'd still 
have to {{readBytes}} one byte at a time since {{RandomAccessInput}} does not 
have a {{readBytes}} method?  But ... maybe {{IndexInput}} would give good 
performance in that case?  We should probably pursue that separately though...

> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Issue Comment Deleted] (LUCENE-5755) Explore alternative build systems

[Vladimir Kroz (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vladimir Kroz updated LUCENE-5755:
--
Comment: was deleted

(was: +1)

> Explore alternative build systems
> -
>
> Key: LUCENE-5755
> URL: https://issues.apache.org/jira/browse/LUCENE-5755
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Priority: Minor
>
> I am dissatisfied with how ANT and submodules currently work in Lucene/ Solr. 
> It's not even the tool's fault; it seems Lucene builds just hit the borders 
> of what it can do, especially in terms of submodule dependencies etc.
> I don't think Maven will help much too, given certain things I'd like to have 
> in the build (for example collect all tests globally for a single execution 
> phase at the end of the build, to support better load-balancing).
> I'd like to explore Gradle as an alternative. This task is a notepad for 
> thoughts and experiments.
> An example of a complex (?) gradle build is javafx, for example.
> http://hg.openjdk.java.net/openjfx/8/master/rt/file/f89b7dc932af/build.gradle



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-5755) Explore alternative build systems

[Vladimir Kroz (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-5755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744532#comment-16744532
 ] 

Vladimir Kroz commented on LUCENE-5755:
---

+1

> Explore alternative build systems
> -
>
> Key: LUCENE-5755
> URL: https://issues.apache.org/jira/browse/LUCENE-5755
> Project: Lucene - Core
>  Issue Type: Task
>Reporter: Dawid Weiss
>Priority: Minor
>
> I am dissatisfied with how ANT and submodules currently work in Lucene/ Solr. 
> It's not even the tool's fault; it seems Lucene builds just hit the borders 
> of what it can do, especially in terms of submodule dependencies etc.
> I don't think Maven will help much too, given certain things I'd like to have 
> in the build (for example collect all tests globally for a single execution 
> phase at the end of the build, to support better load-balancing).
> I'd like to explore Gradle as an alternative. This task is a notepad for 
> thoughts and experiments.
> An example of a complex (?) gradle build is javafx, for example.
> http://hg.openjdk.java.net/openjfx/8/master/rt/file/f89b7dc932af/build.gradle



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744519#comment-16744519
 ] 

Mike Sokolov commented on LUCENE-8635:
--

Right, it seems crazy that makes a difference. I guess there is a tiny bit less 
arithmetic in the RandomAccess version as well. I guess there can be a lot of 
small reads of the terms dictionary

> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-8.x-MacOSX (64bit/jdk1.8.0) - Build # 11 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/11/
Java: 64bit/jdk1.8.0 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.client.solrj.impl.CloudSolrClientTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [SolrCore, 
MockDirectoryWrapper, MockDirectoryWrapper, MockDirectoryWrapper, 
MockDirectoryWrapper] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.core.SolrCore  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.core.SolrCore.(SolrCore.java:1056)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at 
org.apache.solr.core.SolrCore.initSnapshotMetaDataManager(SolrCore.java:510)  
at org.apache.solr.core.SolrCore.(SolrCore.java:961)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:361)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:740)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:969)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.lambda$handleRequestBody$0(CoreAdminHandler.java:188)
  at 
com.codahale.metrics.InstrumentedExecutorService$InstrumentedRunnable.run(InstrumentedExecutorService.java:176)
  at 
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209)
  at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) 
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) 
 at java.lang.Thread.run(Thread.java:748)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 

[jira] [Commented] (SOLR-12373) DocBasedVersionConstraintsProcessor doesn't work when schema has required fields

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744492#comment-16744492
 ] 

Hoss Man commented on SOLR-12373:
-

Tomas: LGTM ... although I would suggest pruning your {{requiredFields}} set of 
any SchemaField were {{null != SchemaField.getDefaultValue()}} to eliminate 
obvious false positive WARN messages.

> DocBasedVersionConstraintsProcessor doesn't work when schema has required 
> fields
> 
>
> Key: SOLR-12373
> URL: https://issues.apache.org/jira/browse/SOLR-12373
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Tomás Fernández Löbbe
>Assignee: Tomás Fernández Löbbe
>Priority: Minor
> Attachments: SOLR-12373.patch, SOLR-12373.patch, SOLR-12373.patch
>
>
> DocBasedVersionConstraintsProcessor creates tombstones when processing a 
> delete by id. Those tombstones only have id (or whatever the unique key name 
> is) and version field(s), however, if the schema defines some required 
> fields, adding the tombstone will fail.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-NightlyTests-master - Build # 1752 - Failure

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1752/

2 tests failed.
FAILED:  junit.framework.TestSuite.org.apache.solr.cloud.LeaderTragicEventTest

Error Message:
ObjectTracker found 1 object(s) that were not released!!! 
[MockDirectoryWrapper] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MockDirectoryWrapper  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:95)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:772)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:969)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:394)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:340)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:164)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) 
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
 at org.eclipse.jetty.server.Server.handle(Server.java:502)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at 
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)  at 
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
  at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at 
org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:411)
  at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:305)  
at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:159)  
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)  at 
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) 
 at java.lang.Thread.run(Thread.java:748)   expected null, but 
was:(SolrCore.java:969)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
 

[jira] [Commented] (SOLR-8291) NPE calling export handler when useFilterForSortedQuery=true

[Rahul Goswami (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-8291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744483#comment-16744483
 ] 

Rahul Goswami commented on SOLR-8291:
-

Facing the same issue on 7.2.1 while using /export handler... Is Ron's patch 
the way to go? 

> NPE calling export handler when useFilterForSortedQuery=true
> 
>
> Key: SOLR-8291
> URL: https://issues.apache.org/jira/browse/SOLR-8291
> Project: Solr
>  Issue Type: Bug
>  Components: SolrCloud
>Affects Versions: 5.2.1
>Reporter: Ray
>Priority: Major
> Attachments: SOLR-8291.patch, solr.log
>
>
> *Updated*: The stacktrace below was created when the solrconfig.xml has the 
> following element:
> {code}
>  true
> {code}
> It was determined that useFilterForSortedQuery is incompatible with the 
> /export handler.
> See the comments near the end of the ticket for a potential work around if 
> this flag needs to be set.
> Get NPE during calling export handler, here is the stack trace:
>   at org.apache.lucene.util.BitSetIterator.(BitSetIterator.java:58)
>   at 
> org.apache.solr.response.SortingResponseWriter.write(SortingResponseWriter.java:138)
>   at 
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:53)
>   at 
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:727)
>   at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:459)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:227)
>   at 
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:196)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:274)
>   at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:242)
>   at 
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:275)
>   at 
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:161)
>   at 
> org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:181)
>   at 
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.event(CatalinaContext.java:285)
>   at 
> org.jboss.modcluster.catalina.CatalinaContext$RequestListenerValve.invoke(CatalinaContext.java:261)
>   at 
> org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:88)
>   at 
> org.jboss.web.tomcat.security.SecurityContextEstablishmentValve.invoke(SecurityContextEstablishmentValve.java:100)
>   at 
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:159)
>   at 
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
>   at 
> org.jboss.web.tomcat.service.jca.CachedConnectionValve.invoke(CachedConnectionValve.java:158)
>   at 
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:567)
>   at 
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
>   at 
> org.jboss.web.tomcat.service.request.ActiveRequestResponseCacheValve.invoke(ActiveRequestResponseCacheValve.java:53)
>   at 
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:362)
>   at 
> org.apache.coyote.ajp.AjpAprProcessor.process(AjpAprProcessor.java:489)
>   at 
> org.apache.coyote.ajp.AjpAprProtocol$AjpConnectionHandler.process(AjpAprProtocol.java:452)
>   at 
> org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:2019)
>   at java.lang.Thread.run(Thread.java:745)
> It seems there are some FixedBitSet was set to null



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13140) Harden SearchRateTriggerIntegrationTest

[Andrzej Bialecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744468#comment-16744468
 ] 

Andrzej Bialecki  commented on SOLR-13140:
--

The internal traffic is between replicas and shard leader. This analysis makes 
sense, and the patch looks good too - indeed, the idea of these tests is to 
detect underutilized replicas and nodes, where search traffic falls below the 
threshold.

Perhaps it would make sense to roll this method and {{testDeleteNode}} into 
one, because it's basically the same as the other methods, only it verifies 
that the trigger can generate a (currently unsupported) DELETENODE request in 
response to the same condition. 

> Harden SearchRateTriggerIntegrationTest
> ---
>
> Key: SOLR-13140
> URL: https://issues.apache.org/jira/browse/SOLR-13140
> Project: Solr
>  Issue Type: Sub-task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Hoss Man
>Assignee: Hoss Man
>Priority: Major
> Attachments: SOLR-13140.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12977) Autoscaling policy initialisation tries to fetch metrics from dead nodes

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744451#comment-16744451
 ] 

Cassandra Targett commented on SOLR-12977:
--

[~noble.paul], This appears in CHANGES.txt for 7.6, so it should be resolved?

> Autoscaling policy initialisation tries to fetch metrics from dead nodes
> 
>
> Key: SOLR-12977
> URL: https://issues.apache.org/jira/browse/SOLR-12977
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>Priority: Minor
> Fix For: 7.6, 8.0
>
>
> Autoscaling policy initialisation tries to fetch metrics for each node during 
> construction. However, it does not skip the known dead nodes causing a 
> timeout to be logged. We should skip such requests entirely.
> {code}
> 20579 WARN  (AutoscalingActionExecutor-37-thread-1) [] 
> o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node 
> 127.0.0.1:63255_solr
> org.apache.solr.client.solrj.SolrServerException: Server refused connection 
> at: http://127.0.0.1:63255/solr
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:650)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1260) 
> ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.invoke(SolrClientNodeStateProvider.java:342)
>  ~[java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:195)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetrics(SolrClientNodeStateProvider.java:186)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getReplicaInfo(SolrClientNodeStateProvider.java:169)
>  [java/:?]
>   at 
> org.apache.solr.client.solrj.cloud.autoscaling.Row.(Row.java:60) 
> [java/:?]
>   at 
> org.apache.solr.client.solrj.cloud.autoscaling.Suggester.getSuggestion(Suggester.java:181)
>  [java/:?]
>   at 
> org.apache.solr.cloud.autoscaling.ComputePlanAction.process(ComputePlanAction.java:114)
>  [java/:?]
>   at 
> org.apache.solr.cloud.autoscaling.ScheduledTriggers.lambda$null$419(ScheduledTriggers.java:308)
>  [java/:?]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[David Smiley (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1675#comment-1675
 ] 

David Smiley commented on SOLR-5211:


{quote}I'm unclear on your meaning. I would think the default behavior when 
deleting a parent (regardless of by id or by query) should also delete its 
children, to avoid ending up with orphans.
{quote}
That would certainly be nice but it's a technical challenge that we don't have 
a solution for.  In this issue we've tackled delete-by-id only (and for 
better/worse it only works on a top/root doc).  If you have ideas on 
delete-by-query then I suggest filing a new issue.

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (SOLR-12908) Add a default set of cluster preferences

[Cassandra Targett (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cassandra Targett resolved SOLR-12908.
--
Resolution: Not A Problem

Marking as Not a Problem. I have some doc changes I will push shortly that will 
make these defaults more clear in the Ref Guide.

> Add a default set of cluster preferences
> 
>
> Key: SOLR-12908
> URL: https://issues.apache.org/jira/browse/SOLR-12908
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Varun Thacker
>Priority: Major
>
> Similar to SOLR-12845 where we want to add a set of default cluster policies, 
> we should add some default cluster preferences as well
>  
> We should always be trying to minimze cores , maximize freedisk for example.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12023) Autoscaling policy engine shuffles replicas needlessly and can also suggest nonexistent replicas to be moved

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744433#comment-16744433
 ] 

Cassandra Targett commented on SOLR-12023:
--

Did the commits here get straightened out? Is this in a release? Can it be 
resolved again?

> Autoscaling policy engine shuffles replicas needlessly and can also suggest 
> nonexistent replicas to be moved
> 
>
> Key: SOLR-12023
> URL: https://issues.apache.org/jira/browse/SOLR-12023
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling, SolrCloud
>Reporter: Shalin Shekhar Mangar
>Assignee: Noble Paul
>Priority: Major
> Fix For: 7.6, 8.0
>
> Attachments: SOLR-11066-failing.patch, SOLR-12023.patch
>
>
> A test that I wrote in SOLR-11066 found the following problem:
> Cluster: 2 nodes
> Collection: 1 shard, 3 replicas, maxShardsPerNode=5
> No autoscaling policy or preference applied
> When the trigger runs, the computed plan needlessly shuffles all three 
> replicas and then proceeds to return suggestions with only numbers as core 
> names. These cores do not exist. I found that these numbers are generated 
> internally by the framework as placeholders for moved cores for further 
> calculations. They should never ever be suggested to the users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12806) when strict=false is specified, prioritize node allocation using non strict rules

[Cassandra Targett (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12806?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744426#comment-16744426
 ] 

Cassandra Targett commented on SOLR-12806:
--

[~noble.paul], so is this fixed? It's in 7.6 CHANGES.txt.

> when strict=false is specified, prioritize node allocation using non strict 
> rules
> -
>
> Key: SOLR-12806
> URL: https://issues.apache.org/jira/browse/SOLR-12806
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: AutoScaling
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Attachments: SOLR-12806.patch
>
>
> if a rule is specified as
> for instance if a policy rule as follows exists
> {code:java}
> {"replica" : "#ALL", "freedisk" : "<500", "strict" : false}
> {code}
>  
> If no no nodes have {{freedisk}} more than 500 GB, Solr ignores this rule 
> completely and assign nodes. Ideally it should still prefer a node with 
> {{freedisk}} of 450GB compared to a node that has {{freedisk}} of 400GB



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-repro - Build # 2685 - Still Unstable

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-repro/2685/

[...truncated 28 lines...]
[repro] Jenkins log URL: 
https://builds.apache.org/job/Lucene-Solr-NightlyTests-master/1751/consoleText

[repro] Revision: d965b3547e2fb87cb7551687bae312a0ff62e526

[repro] Ant options: -Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
[repro] Repro line:  ant test  -Dtestcase=TestStressCloudBlindAtomicUpdates 
-Dtests.method=test_dv -Dtests.seed=BB333F3561AF4F2B -Dtests.multiplier=2 
-Dtests.nightly=true -Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=sr-Latn-RS -Dtests.timezone=Asia/Hong_Kong -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[repro] git rev-parse --abbrev-ref HEAD
[repro] git rev-parse HEAD
[repro] Initial local git branch/revision: 
e35adf6824b091cdf39984a7e426c2e886f56552
[repro] git fetch
[repro] git checkout d965b3547e2fb87cb7551687bae312a0ff62e526

[...truncated 2 lines...]
[repro] git merge --ff-only

[...truncated 1 lines...]
[repro] ant clean

[...truncated 6 lines...]
[repro] Test suites by module:
[repro]solr/core
[repro]   TestStressCloudBlindAtomicUpdates
[repro] ant compile-test

[...truncated 3592 lines...]
[repro] ant test-nocompile -Dtests.dups=5 -Dtests.maxfailures=5 
-Dtests.class="*.TestStressCloudBlindAtomicUpdates" -Dtests.showOutput=onerror 
-Dtests.multiplier=2 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.seed=BB333F3561AF4F2B -Dtests.multiplier=2 -Dtests.nightly=true 
-Dtests.slow=true 
-Dtests.linedocsfile=/home/jenkins/jenkins-slave/workspace/Lucene-Solr-NightlyTests-master/test-data/enwiki.random.lines.txt
 -Dtests.locale=sr-Latn-RS -Dtests.timezone=Asia/Hong_Kong -Dtests.asserts=true 
-Dtests.file.encoding=ISO-8859-1

[...truncated 377640 lines...]
[repro] Setting last failure code to 256

[repro] Failures:
[repro]   3/5 failed: org.apache.solr.cloud.TestStressCloudBlindAtomicUpdates
[repro] git checkout e35adf6824b091cdf39984a7e426c2e886f56552

[...truncated 2 lines...]
[repro] Exiting with code 256

[...truncated 5 lines...]

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Ankit Jain (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744419#comment-16744419
 ] 

Ankit Jain commented on LUCENE-8635:


Thanks [~sokolov] for updating patch and doing another run. As per my 
understanding, seek operation has very less overhead (should be in micro 
seconds), as it just sets the buffer to right position? Maybe the number of 
seek operations is huge and they add up.

> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12903) Query Source Tracker custom Solr component

[Hoss Man (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744353#comment-16744353
 ] 

Hoss Man commented on SOLR-12903:
-

So, my first tought when reviewing this patch is that the entire "qi" param 
concept (with the configure list of allowed "queryIdentifiers" and the 
"failQueries" option to reject requests if the "qi" param isn't specified and 
allowed) really just seems like an inferior version of the the existing 
Authn/Authz Plugin support in solr...

[https://lucene.apache.org/solr/guide/7_6/authentication-and-authorization-plugins.html]

...which has the existing RuleBasedAuthorizationPlugin that can reject/allow 
access to specific request handlers based on the user (ie: "qi") – with the 
added bonus that the users will be authenticated first. (but if there's a 
particular use case where trued authentication isn't needed, then use use the 
empty string as the password for all users. (or I suppose a solr cluster 
_could_ use a custom {{AuthenticationPlugin}} that just trusted any request 
with a "qi" param and used that as the user name w/o requiring any 
password/credentials)

But once you take the "authorizing user access to a /path" out of the picture 
since we already have a good solution for that, what's left is a really 
interesting bit: "recording per user metrics for /path" ... i think that makes 
a lot of sense as an optional SearchComponent. It may not scale well if the 
number of unique users is astronomical (ie: forwarded kerberos credenials from 
a front end application) so it should definitely still be an optional 
component, but for many cases – particularly where client applications each use 
unique credentials to talk to the solr cluster – it would make a lot of sense.

The one hitch is that the way metrics are _typical_ "created" is the way the 
patch currently does things: on init, created a MetricsRegistry with the list 
of pre-defined/hardcoded Metrics you plan on using in your code. But in order 
for this to be general enough to support per-user metrics (w/o making hardcoded 
assumptions about the list of users) that won't work.

I asked [~ab] about this offline, and he pointed out that we already have a 
solution for this type of situation in Solr via the {{MetricsMap implements 
Gauge>}} class. We use this in places like 
{{SolrFieldCacheBean}} where the set of metrics can change over time as solr is 
running, and {{MetricUtils}} has helper functions for unwinding & filtering 
compound/sub metrics (ie: exposing Meters in via MetricsMap). He suggested 
using a SolrCache as a backing store for the MetricsMap – which would also help 
limit the risk of an explosion of users (the the choice between and LRU cache 
or an LFU cache would let solr admins control whether they care more about the 
metrics for the recently active users or the most frequently active users 
if/when an explosion did occur)

So I think what would make sense is:
 * Rename this something like AuthenticatedUserMetricsComponent
 * Make the only required configuration option be the name of an already 
configured user defined {{}}
 ** our docs should strongly encourage using NoOpRegenerator
 * register a MetricsMap with an initializer backed by that cache
 * Rip out all the "qi" logic and replace it with code to just use 
{{SolrQueryRequest.getUserPrincipal()}}
 * use the configured cache to track the username => Meter mapping and record 
the request.

> Query Source Tracker custom Solr component
> --
>
> Key: SOLR-12903
> URL: https://issues.apache.org/jira/browse/SOLR-12903
> Project: Solr
>  Issue Type: Task
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Tirth Rajen Mehta
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Added a Query Source Tracker custom Solr component 
> (https://github.com/apache/lucene-solr/pull/478) :
>  * This component can be configured for a RequestHandler for query requests.
>  * This component mandates that clients to pass in a "qi" request parameter 
> with a valid value which is configured in the SearchComponent definition in 
> the solrconfig.xml file.
>  * It fails the query if the "qi" parameter is missing or if the value passed 
> in is in-valid. This behavior of failing the queries can be controlled by the 
> failQueries config parameter.
>  * It also collects the rate per sec metric per unique "qi" value.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744344#comment-16744344
 ] 

Mike Sokolov commented on LUCENE-8635:
--

Following a suggestion from ~mikemccand I tried a slightly different version of 
this, making use of randomAccessSlice to avoid some calls to seek(), and this 
gives better perf in the benchmarks. I also spent some time trying to 
understand FST's backwards-seeking behavior. Based on my crude understanding, 
and comment from Mike again, it seems as if with some work it would be possible 
to make it more naturally forward-seeking, but it's not obvious that in general 
you would get more local cache-friendly access patterns from that. Still you 
might; probably needs some experimentation to know for sure. Here are the 
benchmark #s from the random-access patch:
{noformat}
Task  QPS before  StdDev   QPS after  StdDev
Pct diff
PKLookup  133.62  (2.2%)  123.74  (1.5%)   
-7.4% ( -10% -   -3%)
  AndHighLow 3411.49  (3.2%) 3268.04  (3.1%)   
-4.2% ( -10% -2%)
BrowseDayOfYearTaxoFacets10067.18  (4.3%) 9828.65  (3.5%)   
-2.4% (  -9% -5%)
 LowTerm 3567.48  (1.2%) 3489.27  (1.7%)   
-2.2% (  -5% -0%)
  Fuzzy1  147.67  (3.1%)  144.65  (2.4%)   
-2.0% (  -7% -3%)
   BrowseMonthTaxoFacets10102.27  (4.2%) 9901.49  (4.1%)   
-2.0% (  -9% -6%)
  Fuzzy2   62.00  (2.8%)   60.87  (2.4%)   
-1.8% (  -6% -3%)
 MedTerm 2694.87  (2.0%) 2647.08  (2.1%)   
-1.8% (  -5% -2%)
  AndHighMed 1171.52  (2.7%) 1154.25  (2.8%)   
-1.5% (  -6% -4%)
HighTerm 2061.53  (2.3%) 2032.84  (2.5%)   
-1.4% (  -6% -3%)
 MedSloppyPhrase  266.60  (3.4%)  263.01  (4.2%)   
-1.3% (  -8% -6%)
  OrHighHigh  278.90  (4.0%)  275.35  (4.7%)   
-1.3% (  -9% -7%)
HighSloppyPhrase  107.68  (5.5%)  106.34  (5.6%)   
-1.2% ( -11% -   10%)
 Respell  118.26  (2.1%)  116.95  (2.2%)   
-1.1% (  -5% -3%)
 AndHighHigh  472.93  (4.4%)  467.78  (3.3%)   
-1.1% (  -8% -6%)
   OrHighMed  755.21  (2.9%)  748.34  (3.3%)   
-0.9% (  -6% -5%)
 MedSpanNear  308.31  (3.3%)  305.59  (3.8%)   
-0.9% (  -7% -6%)
Wildcard  869.37  (3.5%)  862.74  (1.9%)   
-0.8% (  -5% -4%)
   HighTermMonthSort  871.33  (7.1%)  865.80  (6.1%)   
-0.6% ( -12% -   13%)
   MedPhrase  449.39  (3.0%)  446.55  (2.4%)   
-0.6% (  -5% -4%)
 LowSpanNear  391.10  (3.3%)  388.77  (3.8%)   
-0.6% (  -7% -6%)
 LowSloppyPhrase  406.57  (3.8%)  404.23  (3.6%)   
-0.6% (  -7% -7%)
  HighPhrase  239.84  (3.7%)  238.78  (3.3%)   
-0.4% (  -7% -6%)
 Prefix3 1230.56  (5.0%) 1225.52  (2.9%)   
-0.4% (  -7% -7%)
HighSpanNear  107.34  (5.2%)  107.20  (5.3%)   
-0.1% ( -10% -   10%)
   LowPhrase  438.52  (3.4%)  438.14  (2.5%)   
-0.1% (  -5% -5%)
BrowseDateTaxoFacets   11.14  (4.0%)   11.16  (7.0%)
0.2% ( -10% -   11%)
   HighTermDayOfYearSort  606.85  (6.7%)  608.65  (5.4%)
0.3% ( -11% -   13%)
  IntNRQ  987.08 (12.5%)  990.96 (13.5%)
0.4% ( -22% -   30%)
   OrHighLow  553.72  (3.2%)  558.09  (3.5%)
0.8% (  -5% -7%)
BrowseDayOfYearSSDVFacets   38.23  (3.9%)   38.66  (4.1%)
1.1% (  -6% -9%)
   BrowseMonthSSDVFacets   42.05  (3.5%)   42.57  (3.7%)
1.2% (  -5% -8%)

{noformat}

> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required 

[JENKINS] Lucene-Solr-SmokeRelease-8.x - Build # 4 - Still Failing

[Apache Jenkins Server]

Build: https://builds.apache.org/job/Lucene-Solr-SmokeRelease-8.x/4/

No tests ran.

Build Log:
[...truncated 23487 lines...]
[asciidoctor:convert] asciidoctor: ERROR: about-this-guide.adoc: line 1: 
invalid part, must have at least one section (e.g., chapter, appendix, etc.)
[asciidoctor:convert] asciidoctor: ERROR: solr-glossary.adoc: line 1: invalid 
part, must have at least one section (e.g., chapter, appendix, etc.)
 [java] Processed 2466 links (2017 relative) to 3229 anchors in 247 files
 [echo] Validated Links & Anchors via: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr-ref-guide/bare-bones-html/

-dist-changes:
 [copy] Copying 4 files to 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/changes

package:

-unpack-solr-tgz:

-ensure-solr-tgz-exists:
[mkdir] Created dir: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked
[untar] Expanding: 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/package/solr-8.0.0.tgz
 into 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/solr/build/solr.tgz.unpacked

generate-maven-artifacts:

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:
[ivy:configure] :: loading settings :: file = 
/home/jenkins/jenkins-slave/workspace/Lucene-Solr-SmokeRelease-8.x/lucene/top-level-ivy-settings.xml

resolve:

ivy-availability-check:
[loadresource] Do not set property disallowed.ivy.jars.list as its length is 0.

-ivy-fail-disallowed-ivy-version:

ivy-fail:

ivy-configure:

[jira] [Comment Edited] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Mike Sokolov (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744344#comment-16744344
 ] 

Mike Sokolov edited comment on LUCENE-8635 at 1/16/19 6:54 PM:
---

Following a suggestion from [~mikemccand] I tried a slightly different version 
of this, making use of randomAccessSlice to avoid some calls to seek(), and 
this gives better perf in the benchmarks. I also spent some time trying to 
understand FST's backwards-seeking behavior. Based on my crude understanding, 
and comment from Mike again, it seems as if with some work it would be possible 
to make it more naturally forward-seeking, but it's not obvious that in general 
you would get more local cache-friendly access patterns from that. Still you 
might; probably needs some experimentation to know for sure. Here are the 
benchmark #s from the random-access patch:
{noformat}
Task  QPS before  StdDev   QPS after  StdDev
Pct diff
PKLookup  133.62  (2.2%)  123.74  (1.5%)   
-7.4% ( -10% -   -3%)
  AndHighLow 3411.49  (3.2%) 3268.04  (3.1%)   
-4.2% ( -10% -2%)
BrowseDayOfYearTaxoFacets10067.18  (4.3%) 9828.65  (3.5%)   
-2.4% (  -9% -5%)
 LowTerm 3567.48  (1.2%) 3489.27  (1.7%)   
-2.2% (  -5% -0%)
  Fuzzy1  147.67  (3.1%)  144.65  (2.4%)   
-2.0% (  -7% -3%)
   BrowseMonthTaxoFacets10102.27  (4.2%) 9901.49  (4.1%)   
-2.0% (  -9% -6%)
  Fuzzy2   62.00  (2.8%)   60.87  (2.4%)   
-1.8% (  -6% -3%)
 MedTerm 2694.87  (2.0%) 2647.08  (2.1%)   
-1.8% (  -5% -2%)
  AndHighMed 1171.52  (2.7%) 1154.25  (2.8%)   
-1.5% (  -6% -4%)
HighTerm 2061.53  (2.3%) 2032.84  (2.5%)   
-1.4% (  -6% -3%)
 MedSloppyPhrase  266.60  (3.4%)  263.01  (4.2%)   
-1.3% (  -8% -6%)
  OrHighHigh  278.90  (4.0%)  275.35  (4.7%)   
-1.3% (  -9% -7%)
HighSloppyPhrase  107.68  (5.5%)  106.34  (5.6%)   
-1.2% ( -11% -   10%)
 Respell  118.26  (2.1%)  116.95  (2.2%)   
-1.1% (  -5% -3%)
 AndHighHigh  472.93  (4.4%)  467.78  (3.3%)   
-1.1% (  -8% -6%)
   OrHighMed  755.21  (2.9%)  748.34  (3.3%)   
-0.9% (  -6% -5%)
 MedSpanNear  308.31  (3.3%)  305.59  (3.8%)   
-0.9% (  -7% -6%)
Wildcard  869.37  (3.5%)  862.74  (1.9%)   
-0.8% (  -5% -4%)
   HighTermMonthSort  871.33  (7.1%)  865.80  (6.1%)   
-0.6% ( -12% -   13%)
   MedPhrase  449.39  (3.0%)  446.55  (2.4%)   
-0.6% (  -5% -4%)
 LowSpanNear  391.10  (3.3%)  388.77  (3.8%)   
-0.6% (  -7% -6%)
 LowSloppyPhrase  406.57  (3.8%)  404.23  (3.6%)   
-0.6% (  -7% -7%)
  HighPhrase  239.84  (3.7%)  238.78  (3.3%)   
-0.4% (  -7% -6%)
 Prefix3 1230.56  (5.0%) 1225.52  (2.9%)   
-0.4% (  -7% -7%)
HighSpanNear  107.34  (5.2%)  107.20  (5.3%)   
-0.1% ( -10% -   10%)
   LowPhrase  438.52  (3.4%)  438.14  (2.5%)   
-0.1% (  -5% -5%)
BrowseDateTaxoFacets   11.14  (4.0%)   11.16  (7.0%)
0.2% ( -10% -   11%)
   HighTermDayOfYearSort  606.85  (6.7%)  608.65  (5.4%)
0.3% ( -11% -   13%)
  IntNRQ  987.08 (12.5%)  990.96 (13.5%)
0.4% ( -22% -   30%)
   OrHighLow  553.72  (3.2%)  558.09  (3.5%)
0.8% (  -5% -7%)
BrowseDayOfYearSSDVFacets   38.23  (3.9%)   38.66  (4.1%)
1.1% (  -6% -9%)
   BrowseMonthSSDVFacets   42.05  (3.5%)   42.57  (3.7%)
1.2% (  -5% -8%)

{noformat}


was (Author: sokolov):
Following a suggestion from ~mikemccand I tried a slightly different version of 
this, making use of randomAccessSlice to avoid some calls to seek(), and this 
gives better perf in the benchmarks. I also spent some time trying to 
understand FST's backwards-seeking behavior. Based on my crude understanding, 
and comment from Mike again, it seems as if with some work it would be possible 
to make it more naturally forward-seeking, but it's not obvious that in general 
you would get more local cache-friendly access patterns from that. Still you 
might; probably needs some experimentation to know for sure. Here are the 
benchmark #s from the random-access patch:
{noformat}
Task  QPS before  StdDev   QPS after  StdDev  

[jira] [Updated] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

[Mike Sokolov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mike Sokolov updated LUCENE-8635:
-
Attachment: ra.patch

> Lazy loading Lucene FST offheap using mmap
> --
>
> Key: LUCENE-8635
> URL: https://issues.apache.org/jira/browse/LUCENE-8635
> Project: Lucene - Core
>  Issue Type: New Feature
>  Components: core/FSTs
> Environment: I used below setup for es_rally tests:
> single node i3.xlarge running ES 6.5
> es_rally was running on another i3.xlarge instance
>Reporter: Ankit Jain
>Priority: Major
> Attachments: offheap.patch, ra.patch, rally_benchmark.xlsx
>
>
> Currently, FST loads all the terms into heap memory during index open. This 
> causes frequent JVM OOM issues if the term size gets big. A better way of 
> doing this will be to lazily load FST using mmap. That ensures only the 
> required terms get loaded into memory.
>  
> Lucene can expose API for providing list of fields to load terms offheap. I'm 
> planning to take following approach for this:
>  # Add a boolean property fstOffHeap in FieldInfo
>  # Pass list of offheap fields to lucene during index open (ALL can be 
> special keyword for loading ALL fields offheap)
>  # Initialize the fstOffHeap property during lucene index open
>  # FieldReader invokes default FST constructor or OffHeap constructor based 
> on fstOffHeap field
>  
> I created a patch (that loads all fields offheap), did some benchmarks using 
> es_rally and results look good.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-MacOSX (64bit/jdk-9) - Build # 5022 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-MacOSX/5022/
Java: 64bit/jdk-9 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
junit.framework.TestSuite.org.apache.solr.cloud.ChaosMonkeyNothingIsSafeWithPullReplicasTest

Error Message:
ObjectTracker found 5 object(s) that were not released!!! [SolrCore, 
MMapDirectory, InternalHttpClient, MMapDirectory, MMapDirectory] 
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.solr.core.SolrCore  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at org.apache.solr.core.SolrCore.(SolrCore.java:1056)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:394)
  at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:340)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at 
org.apache.solr.client.solrj.embedded.JettySolrRunner$DebugFilter.doFilter(JettySolrRunner.java:164)
  at 
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
  at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540) 
 at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
  at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)  
at 
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
  at 
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
  at 
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)  
at 
org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:703)  
at 
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) 
 at org.eclipse.jetty.server.Server.handle(Server.java:502)  at 
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)  at 
org.eclipse.jetty.server.HttpChannel.run(HttpChannel.java:305)  at 
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
  at 
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683) 
 at java.base/java.lang.Thread.run(Thread.java:844)  
org.apache.solr.common.util.ObjectReleaseTracker$ObjectTrackerException: 
org.apache.lucene.store.MMapDirectory  at 
org.apache.solr.common.util.ObjectReleaseTracker.track(ObjectReleaseTracker.java:42)
  at 
org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:348)
  at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:361)  at 
org.apache.solr.core.SolrCore.initIndex(SolrCore.java:740)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:969)  at 
org.apache.solr.core.SolrCore.(SolrCore.java:876)  at 
org.apache.solr.core.CoreContainer.createFromDescriptor(CoreContainer.java:1189)
  at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1099)  at 
org.apache.solr.handler.admin.CoreAdminOperation.lambda$static$0(CoreAdminOperation.java:92)
  at 
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:360)
  at 
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:395)
  at 
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:180)
  at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
  at org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:736)  
at 
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:717)  
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)  at 

[jira] [Comment Edited] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744256#comment-16744256
 ] 

Thomas Aglassinger edited comment on SOLR-13126 at 1/16/19 5:24 PM:


We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 -> partial match is scored with 0 instead of 2
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]
  -> partial match is scored with 1 instead of 2, which is the current behavior 
on master

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE. The actual number of changed lines per file is 
very small.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.


was (Author: roskakori):
We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, BoostingQuery
|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 -> partial match is scored with 0 instead of 2
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]
  -> partial match is scored with 1 instead of 2, which is the current behavior 
on master

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE. The actual number of changed lines per file is 
very small.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: 
> 0001-use-deprecated-classes-to-fix-regression-introduced-.patch, 
> 0002-SOLR-13126-Added-test-case.patch, debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search 

[jira] [Comment Edited] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744256#comment-16744256
 ] 

Thomas Aglassinger edited comment on SOLR-13126 at 1/16/19 5:23 PM:


We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, BoostingQuery
|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 -> partial match is scored with 0 instead of 2
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]
  -> partial match is scored with 1 instead of 2, which is the current behavior 
on master

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE. The actual number of changed lines per file is 
very small.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.


was (Author: roskakori):
We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE. The actual number of changed lines per file is 
very small.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: 
> 0001-use-deprecated-classes-to-fix-regression-introduced-.patch, 
> 0002-SOLR-13126-Added-test-case.patch, debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search results from queries with multiple 
> multiplicative boosts using the Solr functions {{product()}} and {{query()}} 
> result in a score that 

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744268#comment-16744268
 ] 

Lucky Sharma commented on SOLR-5211:


Yes, that will take 2 calls, I have implemented this with 7.2.1 and 6.6, 
Patch is attached in this  https://issues.apache.org/jira/browse/SOLR-13062.

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744267#comment-16744267
 ] 

Ignacio Vera edited comment on LUCENE-8634 at 1/16/19 5:19 PM:
---

Actually, in the example of the patch, the resulting polygon is invalid (it 
becomes a line). 


was (Author: ivera):
Actually, in the example of the patch, the resulting polygon is invalid (it is 
a line). 

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744267#comment-16744267
 ] 

Ignacio Vera commented on LUCENE-8634:
--

Actually, in the example of the patch, the resulting polygon is invalid (it is 
a line). 

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744266#comment-16744266
 ] 

Ignacio Vera commented on LUCENE-8634:
--

It won't fail as there are no checks for intersection there. So yes, the side 
effect would be lack of accuracy on spatial relationships.

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744256#comment-16744256
 ] 

Thomas Aglassinger edited comment on SOLR-13126 at 1/16/19 5:16 PM:


We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE. The actual number of changed lines per file is 
very small.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.


was (Author: roskakori):
We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: 
> 0001-use-deprecated-classes-to-fix-regression-introduced-.patch, 
> 0002-SOLR-13126-Added-test-case.patch, debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search results from queries with multiple 
> multiplicative boosts using the Solr functions {{product()}} and {{query()}} 
> result in a score that is inconsistent with the one from the debugQuery 
> information. Also only the debug score is correct while the actual search 
> results show a wrong score.
> This seems somewhat similar to the 

[jira] [Comment Edited] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744256#comment-16744256
 ] 

Thomas Aglassinger edited comment on SOLR-13126 at 1/16/19 5:15 PM:


We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.
 # The patches are a bit bigger than the should be due to some automatic white 
space reformatting in the IDE.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.


was (Author: roskakori):
We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: 
> 0001-use-deprecated-classes-to-fix-regression-introduced-.patch, 
> 0002-SOLR-13126-Added-test-case.patch, debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search results from queries with multiple 
> multiplicative boosts using the Solr functions {{product()}} and {{query()}} 
> result in a score that is inconsistent with the one from the debugQuery 
> information. Also only the debug score is correct while the actual search 
> results show a wrong score.
> This seems somewhat similar to the behaviour described in 
> https://issues.apache.org/jira/browse/LUCENE-7132, though this issue has been 
> resolved a while ago.
> A little background: we are using Solr as a 

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[Yosef Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744264#comment-16744264
 ] 

Yosef Brown commented on SOLR-5211:
---

That's one workaround I am planning to try (on 7.5) -- delete the parent & 
children, then re-add.  Unfortunately, I believe this will take two calls.

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Aglassinger updated SOLR-13126:
--
Attachment: 0002-SOLR-13126-Added-test-case.patch
0001-use-deprecated-classes-to-fix-regression-introduced-.patch

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: 
> 0001-use-deprecated-classes-to-fix-regression-introduced-.patch, 
> 0002-SOLR-13126-Added-test-case.patch, debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search results from queries with multiple 
> multiplicative boosts using the Solr functions {{product()}} and {{query()}} 
> result in a score that is inconsistent with the one from the debugQuery 
> information. Also only the debug score is correct while the actual search 
> results show a wrong score.
> This seems somewhat similar to the behaviour described in 
> https://issues.apache.org/jira/browse/LUCENE-7132, though this issue has been 
> resolved a while ago.
> A little background: we are using Solr as a search platform for the 
> e-commerce framework SAP Hybris. There the shop administrator can create 
> multiplicative boost rules (see below for an example) where a value like 2.0 
> means that an item gets boosted to 200%. This works fine in the demo shop 
> distributed by SAP but breaks in our shop. We encountered the issue when 
> Upgrading from Solr 7.2.1 / Hybris 6.7 to Solr 7.5 / Hybris 18.8.3 (which 
> would have been named Hybris 6.8 but the version naming schema changed).
> We reduced the Solr query generated by Hybris to the relevant parts and could 
> reproduce the issue in the Solr admin without any Hybris connection.
> I attached the JSON result of a test query but here's a description of the 
> parts that seemed most relevant to me.
> The {{responseHeader.params}} reads (slightly rearranged):
> {code:java}
> "q":"{!boost b=$ymb}(+{!lucene v=$yq})",
> "ymb":"product(query({!v=\"name_text_de\\:Netzteil\\^=2.0\"},1),query({!v=\"name_text_de\\:Sony\\^=3.0\"},1))",
> "yq":"*:*",
> "sort":"score desc",
> "debugQuery":"true",
> // Added to keep the output small but probably unrelated to the actual issue
> "fl":"score,id,code_string,name_text_de",
> "fq":"catalogId:\"someProducts\"",
> "rows":"10",
> {code}
> This example boosts the German product name (field {{name_text_de}}) in case 
> in contains certain terms:
>  * "Netzteil" (power supply) is boosted to 200%
>  * "Sony" is boosted to 300%
> Consequently a product containing both terms should be boosted to 600%.
> Also the query function has the value 1 specified as default in case the name 
> does not contain the respective term resulting in a pseudo boost that 
> preserves the score.
> According to the debug information the parser used is the LuceneQParser, 
> which translates this to the following parsed query:
> {quote}FunctionScoreQuery(FunctionScoreQuery(+*:*, scored by 
> boost(product(query((ConstantScore(name_text_de:netzteil))^2.0,def=1.0),query((ConstantScore(name_text_de:sony))^3.0,def=1.0)
> {quote}
> And the translated boost is:
> {quote}org.apache.lucene.queries.function.valuesource.ProductFloatFunction:product(query((ConstantScore(name_text_de:netzteil))^2.0,def=1.0),query((ConstantScore(name_text_de:sony))^3.0,def=1.0))
> {quote}
> When taking a look at the search result, among other the following products 
> are included (see the JSON comments for an analysis of each result):
> {code:javascript}
>  {
> "id":"someProducts/Online/test711",
> "name_text_de":"Original Sony Vaio Netzteil",
> "code_string":"test711",
> // CORRECT, both "Netzteil" and "Sony" are included in the name
> "score":6.0},
>   {
> "id":"someProducts/Online/taxTestingProductThree",
> "name_text_de":"Steuertestprodukt Zwei",
> "code_string":"taxTestingProductThree",
> // CORRECT, neither "Netzteil" nor "Sony" are included in the name
> "score":1.0},
>   {
> "id":"someProducts/Online/79785630",
> "name_text_de":"GS-Netzteil 20W schwarz",
> "code_string":"79785630",

[jira] [Commented] (SOLR-13126) Inconsistent score in debug and result with multiple multiplicative boosts

[Thomas Aglassinger (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744256#comment-16744256
 ] 

Thomas Aglassinger commented on SOLR-13126:
---

We digged in further and seem to have found the culprit. The test case in the 
attached patch {{0002-SOLR-13126-Added-test-case.patch}} reproduces the bug. 
The last working version is Solr 7.2.1.

Using {{git bisect}} we found out that the issue got introduced with 
LUCENE-8099 (a refactoring). There's two changes that break the scoring in 
different ways:
 * [LUCENE-8099: Deprecate CustomScoreQuery, BoostedQuery, 
BoostingQuery|https://github.com/apache/lucene-solr/commit/b01e6023e1cd3c62260b38c05c8d145ba143a2ac]
 * [LUCENE-8099: Replace BoostQParserPlugin.boostQuery() with 
FunctionScoreQuery.boostByValue()|https://github.com/apache/lucene-solr/commit/0744fea821366a853b8e239e766b9786ef96cb27]

The attached patch 
{{0001-use-deprecated-classes-to-fix-regression-introduced-.patch}} includes an 
experimental fix by reverting some parts of the code to its previous version 
based on a deprecated class the refactoring of LUCENE-8099 tried to replace 
(among other things).

This is a rough initial patch with the following known issues:
 # The patch goes towards solr 7.5.0. This is the version we currently 
experience the issues with and attempt to get back to working for production 
use. Ideally of course the patch would go towards the master and then merged 
back to earlier versions.
 # The fix uses a deprecated class. Ideally it would fix the refactored classes 
from LUCENE-8099.

Nevertheless the test case is generic enough to run on all branches, including 
the current master.

> Inconsistent score in debug and result with multiple multiplicative boosts
> --
>
> Key: SOLR-13126
> URL: https://issues.apache.org/jira/browse/SOLR-13126
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: search
>Affects Versions: 7.5.0
> Environment: Reproduced with macOS 10.14.1, a quick test with Windows 
> 10 showed the same result.
>Reporter: Thomas Aglassinger
>Priority: Major
> Attachments: debugQuery.json, 
> solr_match_neither_nextteil_nor_sony.json, 
> solr_match_neither_nextteil_nor_sony.txt, solr_match_netzteil_and_sony.json, 
> solr_match_netzteil_and_sony.txt, solr_match_netzteil_only.json, 
> solr_match_netzteil_only.txt
>
>
> Under certain circumstances search results from queries with multiple 
> multiplicative boosts using the Solr functions {{product()}} and {{query()}} 
> result in a score that is inconsistent with the one from the debugQuery 
> information. Also only the debug score is correct while the actual search 
> results show a wrong score.
> This seems somewhat similar to the behaviour described in 
> https://issues.apache.org/jira/browse/LUCENE-7132, though this issue has been 
> resolved a while ago.
> A little background: we are using Solr as a search platform for the 
> e-commerce framework SAP Hybris. There the shop administrator can create 
> multiplicative boost rules (see below for an example) where a value like 2.0 
> means that an item gets boosted to 200%. This works fine in the demo shop 
> distributed by SAP but breaks in our shop. We encountered the issue when 
> Upgrading from Solr 7.2.1 / Hybris 6.7 to Solr 7.5 / Hybris 18.8.3 (which 
> would have been named Hybris 6.8 but the version naming schema changed).
> We reduced the Solr query generated by Hybris to the relevant parts and could 
> reproduce the issue in the Solr admin without any Hybris connection.
> I attached the JSON result of a test query but here's a description of the 
> parts that seemed most relevant to me.
> The {{responseHeader.params}} reads (slightly rearranged):
> {code:java}
> "q":"{!boost b=$ymb}(+{!lucene v=$yq})",
> "ymb":"product(query({!v=\"name_text_de\\:Netzteil\\^=2.0\"},1),query({!v=\"name_text_de\\:Sony\\^=3.0\"},1))",
> "yq":"*:*",
> "sort":"score desc",
> "debugQuery":"true",
> // Added to keep the output small but probably unrelated to the actual issue
> "fl":"score,id,code_string,name_text_de",
> "fq":"catalogId:\"someProducts\"",
> "rows":"10",
> {code}
> This example boosts the German product name (field {{name_text_de}}) in case 
> in contains certain terms:
>  * "Netzteil" (power supply) is boosted to 200%
>  * "Sony" is boosted to 300%
> Consequently a product containing both terms should be boosted to 600%.
> Also the query function has the value 1 specified as default in case the name 
> does not contain the respective term resulting in a pseudo boost that 
> preserves the score.
> According to the debug information the parser used is the LuceneQParser, 
> which translates this to the following parsed 

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744233#comment-16744233
 ] 

Adrien Grand commented on LUCENE-8634:
--

I think it depends what happens in such a case: would we fail the query with an 
exception or would we only fail to match where edges cross or are too close to 
each other? The latter sounds acceptable to me due to the documented accuracy 
limitations, but failing a query with a valid polygon less so?

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-12955) Refactor DistributedUpdateProcessor

[Bar Rotstein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SOLR-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bar Rotstein updated SOLR-12955:

Attachment: SOLR-12955.patch

> Refactor DistributedUpdateProcessor
> ---
>
> Key: SOLR-12955
> URL: https://issues.apache.org/jira/browse/SOLR-12955
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Bar Rotstein
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-12955-no-commit.patch, SOLR-12955.patch, 
> SOLR-12955.patch, SOLR-12955.patch
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Lately As I was skimming through Solr's code base I noticed that 
> DistributedUpdateProcessor has a lot of nested if else statements, which 
> hampers code readability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-12955) Refactor DistributedUpdateProcessor

[Bar Rotstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-12955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744229#comment-16744229
 ] 

Bar Rotstein commented on SOLR-12955:
-

Uploaded a new patch file,
 based on the changes made in PR 
[#528|https://github.com/apache/lucene-solr/pull/528].

> Refactor DistributedUpdateProcessor
> ---
>
> Key: SOLR-12955
> URL: https://issues.apache.org/jira/browse/SOLR-12955
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Bar Rotstein
>Assignee: David Smiley
>Priority: Major
> Attachments: SOLR-12955-no-commit.patch, SOLR-12955.patch, 
> SOLR-12955.patch, SOLR-12955.patch
>
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> Lately As I was skimming through Solr's code base I noticed that 
> DistributedUpdateProcessor has a lot of nested if else statements, which 
> hampers code readability.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-13007) Use javabin instead of JSON to send messages to overseer

[Bar Rotstein (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744225#comment-16744225
 ] 

Bar Rotstein commented on SOLR-13007:
-

{quote}This is a micro optimization. Not very essential and it is not very 
urgent{quote}
Is this ticket up for grabs?
If you have not started working on this yet,
I would be more than happy to give this a shot.

> Use javabin instead of JSON to send messages to overseer
> 
>
> Key: SOLR-13007
> URL: https://issues.apache.org/jira/browse/SOLR-13007
> Project: Solr
>  Issue Type: Improvement
>  Security Level: Public(Default Security Level. Issues are Public) 
>Reporter: Noble Paul
>Priority: Major
>
> The messages themselves are ephemeral and the readability is not a big issue. 
> Using javabin can:
> * reduce the payload size
> * make processing faster



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Re: Admin UI - Collections Management

[Branham, Jeremy (Experis)]

Thanks Jan –
I agree it makes sense to have the operation in the collection -> … -> replica 
area, since it’s a related function.
On the other hand, when I’m looking to move replicas around, I’m interested in 
seeing the replicas grouped by server rather than shard.
Seems like the overlap may be hard to avoid without some redesign.

I’d love to contribute to a redesign, or bringing the collection/shard 
functionality into the nodes section. But I’d need to moonlight it, since it 
might be difficult to justify on my employer’s time.

Jeremy Branham
jb...@allstate.com

From: Jan Høydahl 
Reply-To: "dev@lucene.apache.org" 
Date: Wednesday, January 16, 2019 at 7:19 AM
To: Lucene Dev 
Subject: [External] Re: Admin UI - Collections Management

I think perhaps this belongs in the Collections -> coll_name -> Shards -> 
Replicas screen, where you can already delete and add a replica to a shard. 
That screen has a dropdown to select what node the new replica should be 
created on. So if you add a MOVE button next to the DELETE button, and show a 
dropdown with list of nodes, then you should have enough info to construct a 
MOVEREPLICA cmd.

I know that there is a huge overlap between the Collections and Cloud menus, so 
at some point a redesign is needed. Of course we could also add a 
context-sensitive popup menu with a MOVE button to the Cloud->Nodes screen 
where we list replicas...

--
Jan Høydahl, search solution architect
Cominvent AS - 
www.cominvent.com


15. jan. 2019 kl. 19:33 skrev Erick Erickson 
mailto:erickerick...@gmail.com>>:

bq: Thoughts?

I don't do UI work so I just sit back and cheer ;)

Then I do, of course have thoughts...

Mostly, when you're thinking about this think about scale. For
instance, the graph view is nice for a few nodes/collections. 1,000
collections (yes, some people scale that way) with even 2x2
collections is almost, but not quite totally, unusable. Maybe
copy/paste or cut/paste? Or some way to restrict what's shown to the
relevant nodes? More generally how to navigate around when there are
zillions of replicas

Best,
Erick


On Tue, Jan 15, 2019 at 8:39 AM Branham, Jeremy (Experis)
mailto:jb...@allstate.com>> wrote:


Thanks Erick –
I used the MOVEREPLICA command to shuffle them around, but like you mentioned, 
I had to look up nodes names and such.
I’m on the same page; drag-n-drop replica organization.

I was looking at the newest section in the admin-ui  “Cloud > Nodes”
Maybe we could add the drag-n-drop functionality to that last column, where the 
replicas are listed by node. I think we have all the data required in that view 
to construct a MOVEREPLICA command.

Thoughts?


Jeremy Branham
jb...@allstate.com

On 1/14/19, 4:55 PM, "Erick Erickson"  wrote:

   bq. The collections API allowed me to move the replicas around by
   investigating the core names and locations,

   What about the MOVEREPLICA command?
   
https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_6-5F6_collections-2Dapi.html=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=037mwmgIodjUnHNITRxTA5FJu2QXK_AYmNgd-oCyfxU=

   Although that still requires you to know things like node names and
   replica names. It'd be way cool to have
   some kind of drag-n-drop or wizard-driven capability, perhaps built on
   top of that. In addition to Jan's
   suggestions.

   Best,
   Erick

   On Mon, Jan 14, 2019 at 2:37 PM Jan Høydahl  wrote:


Hi and thanks for offering to help.

If you are not familiar with the new Autoscaling framework, I'd start by 
exploring it, since it aims at solving replica placement without explicit 
commands.
https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F6_solrcloud-2Dautoscaling.html=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=SRn57bErK14_HIBo5zPxDGt8KgkslZhmB2zNNCd2jXc=

Next, I'd try to find an open JIRA issue to solve, perhaps something related to 
Collections API and/or Admin UI. Pick something very simple, just to get 
started with the procedure of contributing, and then look at e.g. 
https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D10209=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=XB6iMPC0x1pEBUc-i-g8TwI9Y9QuQHcT87S-0Qx3NV8=
 which seems related in that it aims to expose collection api through UI

--
Jan Høydahl, search solution architect
Cominvent AS - 

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744211#comment-16744211
 ] 

Ignacio Vera commented on LUCENE-8634:
--

Mostly, the Morton optimisation order the points on the encoding space, but not 
sure how relevant it is. Anyway, this should only be relevant if you are 
defining your polygons with centimetre accuracy which it is probably not 
realistic.

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744208#comment-16744208
 ] 

Lucky Sharma commented on SOLR-5211:


What if we delete the complete block and create the block again and add again 
via UpdateChainProcessorFactory. In that way, will update only those 
fields/docs what we want to and put the block back in place

> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744180#comment-16744180
 ] 

Adrien Grand commented on LUCENE-8634:
--

bq. This is the same case as in the tessellator

The tessellator is a bit different because it tessellates into triangles before 
moving to the encoded space? The only downside is that triangles might change 
orientation through encoding, which is not a problem now because the encoding 
doesn't make any assumption about the encoding of triangles?

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8639) SeqNo accounting in IW is broken if many threads start indexing while we flush.

[Simon Willnauer (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Simon Willnauer resolved LUCENE-8639.
-
   Resolution: Fixed
Fix Version/s: master (9.0)
   7.7
   8.0

> SeqNo accounting in IW is broken if many threads start indexing while we 
> flush.
> ---
>
> Key: LUCENE-8639
> URL: https://issues.apache.org/jira/browse/LUCENE-8639
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Simon Willnauer
>Priority: Major
> Fix For: 8.0, 7.7, master (9.0)
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> While this is rare in the wild we have a test failure that shows that our 
> seqNo accounting is broken  when we carry over seqNo to a new delete queue. 
> We had this test-failure:
> {noformat}
> 6:06:08[junit4] Suite: org.apache.lucene.index.TestIndexTooManyDocs
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-8,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=7 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-9,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=6 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 
> 16:06:08[junit4]   2> ??? 14, 2019 11:05:45 ? 
> com.carrotsearch.randomizedtesting.ThreadLeakControl$2 evaluate
> 16:06:08[junit4]   2> WARNING: Suite execution timed out: 
> org.apache.lucene.index.TestIndexTooManyDocs
> 16:06:08[junit4]   2>1) Thread[id=20, 
> name=SUITE-TestIndexTooManyDocs-seed#[43B7C75B765AFEBD], state=RUNNABLE, 
> group=TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> at 
> 

[jira] [Commented] (LUCENE-8639) SeqNo accounting in IW is broken if many threads start indexing while we flush.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744168#comment-16744168
 ] 

ASF subversion and git services commented on LUCENE-8639:
-

Commit 0d306b40b77ba85d1602c60621e93c5b190b6ef9 in lucene-solr's branch 
refs/heads/branch_7x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=0d306b4 ]

LUCENE-8639: Prevent new threadstates from being created while we cut over to a 
new delete queue (#535)

This prevents an edge case where suddenly a lot of threads start indexing
while we carry over sequence ids from the previous to the new delete queue.
We now lock creation of new thread states for a very short time until we 
created and assigned
a new delete queue.

> SeqNo accounting in IW is broken if many threads start indexing while we 
> flush.
> ---
>
> Key: LUCENE-8639
> URL: https://issues.apache.org/jira/browse/LUCENE-8639
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> While this is rare in the wild we have a test failure that shows that our 
> seqNo accounting is broken  when we carry over seqNo to a new delete queue. 
> We had this test-failure:
> {noformat}
> 6:06:08[junit4] Suite: org.apache.lucene.index.TestIndexTooManyDocs
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-8,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=7 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-9,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=6 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   

[jira] [Commented] (LUCENE-8639) SeqNo accounting in IW is broken if many threads start indexing while we flush.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744165#comment-16744165
 ] 

ASF subversion and git services commented on LUCENE-8639:
-

Commit 59ba68697060b48d147b9e118ed946545932d491 in lucene-solr's branch 
refs/heads/branch_8x from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=59ba686 ]

LUCENE-8639: Prevent new threadstates from being created while we cut over to a 
new delete queue (#535)

This prevents an edge case where suddenly a lot of threads start indexing
while we carry over sequence ids from the previous to the new delete queue.
We now lock creation of new thread states for a very short time until we 
created and assigned
a new delete queue.

> SeqNo accounting in IW is broken if many threads start indexing while we 
> flush.
> ---
>
> Key: LUCENE-8639
> URL: https://issues.apache.org/jira/browse/LUCENE-8639
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> While this is rare in the wild we have a test failure that shows that our 
> seqNo accounting is broken  when we carry over seqNo to a new delete queue. 
> We had this test-failure:
> {noformat}
> 6:06:08[junit4] Suite: org.apache.lucene.index.TestIndexTooManyDocs
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-8,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=7 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-9,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=6 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744162#comment-16744162
 ] 

Ignacio Vera commented on LUCENE-8634:
--

{quote}Are there concerns regarding the fact that a polygon might become 
invalid through encoding (and eg. cross itself).
{quote}
The encoding is very fine grid with cells of size 8.39e-8 degrees X 8.39e-8 
degrees. So when moving to the encoding space it can happen that two points 
become the same point or if close enough to an edge in theory it can cross it. 
This is the same case as in the tessellator and never happen during the test. I 
check and Polygon2D can handle when having shared vertex and could not really 
reproduce the other case, so I am not too worry about this.
{quote}Should we keep lines and polygons in the encoded space like boxes?
{quote}
I have been wondering that all the time (It might help performance) but it is a 
bigger change. Sure if we move query shapes to the encoding space, then we 
should probably keep them there.

 

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8639) SeqNo accounting in IW is broken if many threads start indexing while we flush.

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744159#comment-16744159
 ] 

ASF subversion and git services commented on LUCENE-8639:
-

Commit e35adf6824b091cdf39984a7e426c2e886f56552 in lucene-solr's branch 
refs/heads/master from Simon Willnauer
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e35adf6 ]

LUCENE-8639: Prevent new threadstates from being created while we cut over to a 
new delete queue (#535)

This prevents an edge case where suddenly a lot of threads start indexing
while we carry over sequence ids from the previous to the new delete queue.
We now lock creation of new thread states for a very short time until we 
created and assigned
a new delete queue.

> SeqNo accounting in IW is broken if many threads start indexing while we 
> flush.
> ---
>
> Key: LUCENE-8639
> URL: https://issues.apache.org/jira/browse/LUCENE-8639
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Simon Willnauer
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> While this is rare in the wild we have a test failure that shows that our 
> seqNo accounting is broken  when we carry over seqNo to a new delete queue. 
> We had this test-failure:
> {noformat}
> 6:06:08[junit4] Suite: org.apache.lucene.index.TestIndexTooManyDocs
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-8,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=7 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 
> 16:06:08[junit4]   2> ??? 14, 2019 9:05:46 ? 
> com.carrotsearch.randomizedtesting.RandomizedRunner$QueueUncaughtExceptionsHandler
>  uncaughtException
> 16:06:08[junit4]   2> WARNING: Uncaught exception in thread: 
> Thread[Thread-9,5,TGRP-TestIndexTooManyDocs]
> 16:06:08[junit4]   2> java.lang.AssertionError: seqNo=6 vs maxSeqNo=6
> 16:06:08[junit4]   2> at 
> __randomizedtesting.SeedInfo.seed([43B7C75B765AFEBD]:0)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.getNextSequenceNumber(DocumentsWriterDeleteQueue.java:482)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:168)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterDeleteQueue.add(DocumentsWriterDeleteQueue.java:146)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.finishDocument(DocumentsWriterPerThread.java:362)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:264)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:494)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586)
> 16:06:08[junit4]   2> at 
> org.apache.lucene.index.TestIndexTooManyDocs.lambda$testIndexTooManyDocs$0(TestIndexTooManyDocs.java:70)
> 16:06:08[junit4]   2> at java.lang.Thread.run(Thread.java:748)
> 16:06:08[junit4]   2> 

[GitHub] s1monw merged pull request #535: LUCENE-8639: Prevent new threadstates from being created while we cut over to a new delete queue

[GitBox]

s1monw merged pull request #535: LUCENE-8639: Prevent new threadstates from 
being created while we cut over to a new delete queue
URL: https://github.com/apache/lucene-solr/pull/535
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5211) updating parent as childless makes old children orphans

[Yosef Brown (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-5211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744145#comment-16744145
 ] 

Yosef Brown commented on SOLR-5211:
---

[~dsmiley] 
bq.  a delete-by-id would also only delete a root document ... my take-away 
message to this is that we don't need to think of the delete-by-id limitation 
of root documents as a problem. It's unusual to want to do otherwise (and 
error-prone if you leave grandchildren orphaned), and it's possible to do this 
via delete-by-query.

I'm unclear on your meaning.  I would think the default behavior when deleting 
a parent (regardless of by id or by query) should also delete its children, to 
avoid ending up with orphans.  Something like, deleting the block's root 
deletes the whole block.  This is similar to the current functionality that 
overwrites (or rewrites) the whole block, including any children documents, 
when updating a document.


> updating parent as childless makes old children orphans
> ---
>
> Key: SOLR-5211
> URL: https://issues.apache.org/jira/browse/SOLR-5211
> Project: Solr
>  Issue Type: Sub-task
>  Components: update
>Affects Versions: 4.5, 6.0
>Reporter: Mikhail Khludnev
>Assignee: David Smiley
>Priority: Blocker
> Fix For: 8.0
>
> Attachments: SOLR-5211.patch, SOLR-5211.patch, SOLR-5211.patch, 
> SOLR-5211.patch, SOLR-5211.patch
>
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> if I have parent with children in the index, I can send update omitting 
> children. as a result old children become orphaned. 
> I suppose separate \_root_ fields makes much trouble. I propose to extend 
> notion of uniqueKey, and let it spans across blocks that makes updates 
> unambiguous.  
> WDYT? Do you like to see a test proves this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8644) Pass token types to QueryBuilder.newSynonymQuery()

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744132#comment-16744132
 ] 

Alan Woodward commented on LUCENE-8644:
---

Multi-term synonyms go through analyzeGraphPhrase and so don't use 
newSynonymQuery.  Good point on the duplicate terms though, I'll update the 
patch.

> Pass token types to QueryBuilder.newSynonymQuery()
> --
>
> Key: LUCENE-8644
> URL: https://issues.apache.org/jira/browse/LUCENE-8644
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8644.patch
>
>
> SynonymGraphFilter distinguishes between input tokens and tokens that it 
> injects into the token stream by setting the TypeAttribute to SYNONYM.  
> QueryBuilder currently ignores this information, but it would be useful to 
> pass it through to the newSynonymQuery() method, as this would allow 
> queryparsers to apply differential boosts to synonyms and original terms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] mikemccand commented on issue #535: LUCENE-8639: Prevent new threadstates from being created while we cut over to a new delete queue

[GitBox]

mikemccand commented on issue #535: LUCENE-8639: Prevent new threadstates from 
being created while we cut over to a new delete queue
URL: https://github.com/apache/lucene-solr/pull/535#issuecomment-454811616
 
 
   I like the test case!!


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-7.x-Linux (64bit/jdk1.8.0_172) - Build # 3409 - Unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/3409/
Java: 64bit/jdk1.8.0_172 -XX:+UseCompressedOops -XX:+UseSerialGC

1 tests failed.
FAILED:  
org.apache.solr.cloud.TestCloudSearcherWarming.testRepFactor1LeaderStartup

Error Message:
No live SolrServers available to handle this request

Stack Trace:
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available 
to handle this request
at 
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:461)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:884)
at 
org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:817)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:194)
at 
org.apache.solr.client.solrj.SolrRequest.process(SolrRequest.java:211)
at 
org.apache.solr.cloud.MiniSolrCloudCluster.deleteAllCollections(MiniSolrCloudCluster.java:517)
at 
org.apache.solr.cloud.TestCloudSearcherWarming.tearDown(TestCloudSearcherWarming.java:78)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 

[jira] [Commented] (LUCENE-8633) Remove term weighting from interval scoring

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744076#comment-16744076
 ] 

ASF subversion and git services commented on LUCENE-8633:
-

Commit 18c3986cc4bdb38949fae14bd771621345d41776 in lucene-solr's branch 
refs/heads/branch_8x from Alan Woodward
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=18c3986 ]

LUCENE-8633: Remove term weighting from IntervalQuery scores


> Remove term weighting from interval scoring
> ---
>
> Key: LUCENE-8633
> URL: https://issues.apache.org/jira/browse/LUCENE-8633
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8633.patch, LUCENE-8633.patch
>
>
> IntervalScorer currently uses the same scoring mechanism as SpanScorer, 
> summing the IDF of all possibly matching terms from its parent 
> IntervalsSource and using that in conjunction with a sloppy frequency to 
> produce a similarity-based score.  This doesn't really make sense, however, 
> as it means that terms that don't appear in a document can still contribute 
> to the score, and appears to make scores from interval queries comparable 
> with scores from term or phrase queries when they really aren't.
> I'd like to explore a different scoring mechanism for intervals, based purely 
> on sloppy frequency and ignoring term weighting.  This should make the scores 
> easier to reason about, as well as making them useful for things like 
> proximity boosting on boolean queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Resolved] (LUCENE-8633) Remove term weighting from interval scoring

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward resolved LUCENE-8633.
---
   Resolution: Fixed
Fix Version/s: master (9.0)
   8.0

> Remove term weighting from interval scoring
> ---
>
> Key: LUCENE-8633
> URL: https://issues.apache.org/jira/browse/LUCENE-8633
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Fix For: 8.0, master (9.0)
>
> Attachments: LUCENE-8633.patch, LUCENE-8633.patch
>
>
> IntervalScorer currently uses the same scoring mechanism as SpanScorer, 
> summing the IDF of all possibly matching terms from its parent 
> IntervalsSource and using that in conjunction with a sloppy frequency to 
> produce a similarity-based score.  This doesn't really make sense, however, 
> as it means that terms that don't appear in a document can still contribute 
> to the score, and appears to make scores from interval queries comparable 
> with scores from term or phrase queries when they really aren't.
> I'd like to explore a different scoring mechanism for intervals, based purely 
> on sloppy frequency and ignoring term weighting.  This should make the scores 
> easier to reason about, as well as making them useful for things like 
> proximity boosting on boolean queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8633) Remove term weighting from interval scoring

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744077#comment-16744077
 ] 

ASF subversion and git services commented on LUCENE-8633:
-

Commit a8266492416dbc2fcf665983d1cac141e50fa9eb in lucene-solr's branch 
refs/heads/master from Alan Woodward
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a826649 ]

LUCENE-8633: Remove term weighting from IntervalQuery scores


> Remove term weighting from interval scoring
> ---
>
> Key: LUCENE-8633
> URL: https://issues.apache.org/jira/browse/LUCENE-8633
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8633.patch, LUCENE-8633.patch
>
>
> IntervalScorer currently uses the same scoring mechanism as SpanScorer, 
> summing the IDF of all possibly matching terms from its parent 
> IntervalsSource and using that in conjunction with a sloppy frequency to 
> produce a similarity-based score.  This doesn't really make sense, however, 
> as it means that terms that don't appear in a document can still contribute 
> to the score, and appears to make scores from interval queries comparable 
> with scores from term or phrase queries when they really aren't.
> I'd like to explore a different scoring mechanism for intervals, based purely 
> on sloppy frequency and ignoring term weighting.  This should make the scores 
> easier to reason about, as well as making them useful for things like 
> proximity boosting on boolean queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-13141) replicationFactor param cause problems with CDCR

[Krzysztof Watral (JIRA)]

[ 
https://issues.apache.org/jira/browse/SOLR-13141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743816#comment-16743816
 ] 

Krzysztof Watral edited comment on SOLR-13141 at 1/16/19 2:10 PM:
--

problem can be related with SOLR-12998 (same errors in logs), but in current 
case it's not a race condition


was (Author: kwatral):
problem can be related with SOLR-12998 (same errors in logs), but in this case, 
this is not a race condition

> replicationFactor param cause problems with CDCR
> 
>
> Key: SOLR-13141
> URL: https://issues.apache.org/jira/browse/SOLR-13141
> Project: Solr
>  Issue Type: Bug
>  Security Level: Public(Default Security Level. Issues are Public) 
>  Components: CDCR
>Affects Versions: 7.5, 7.6
> Environment: This is system independent problem - exists on windows 
> and linux - reproduced by independent developers
>Reporter: Krzysztof Watral
>Priority: Critical
> Attachments: type 1 - replication wasnt working at all.txt, type 2 - 
> only few documents were being replicated.txt
>
>
> i have encountered some problems with CDCR that are related to the value of 
> {{replicationFactor}} param.
> I ran the solr cloud on two datacenters with 2 nodes on each:
>  * dca:
>  ** dca_node_1
>  ** dca_node_2
>  * dcb
>  ** dcb_node_1
>  ** dcb_node_2
> Then in sequence:
>  * I configured the CDCR on copy of *_default* config set named 
> *_default_cdcr*
>  * I created collection "customer" on both DC from *_default_cdcr* config set 
> with the following parameters:
>  ** {{numShards}} = 2
>  ** {{maxShardsPerNode}} = 2
>  ** {{replicationFactor}} = 2
>  * I disabled cdcr buffer on collections
>  * I ran CDCR on both DC
> CDCR has started without errors in logs. During indexation I have encountered 
> problem [^type 2 - only few documents were being replicated.txt], restart 
> didn't help (documents has not been synchronized between DC )
> Then:
>  * I stopped CDCR on both DC
>  * I stopped all solr nodes
>  * I restarted zookeepers on both DC
>  * I started all solr nodes one by one
>  * few minutes later I stared CDCR on both DC
>  * CDCR has starded with errors (replication between DC is not working) - 
> [^type 1 - replication wasnt working at all.txt]
> {panel}
> I've also discovered that problems appears only in case, when the 
> {{replicationFactor}} parameter is higher than one
> {panel}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[GitHub] mikemccand commented on issue #535: LUCENE-8639: Prevent new threadstates from being created while we cut over to a new delete queue

[GitBox]

mikemccand commented on issue #535: LUCENE-8639: Prevent new threadstates from 
being created while we cut over to a new delete queue
URL: https://github.com/apache/lucene-solr/pull/535#issuecomment-454787650
 
 
   LGTM -- sneaky.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744039#comment-16744039
 ] 

Lucky Sharma edited comment on LUCENE-8640 at 1/16/19 1:30 PM:
---

yes It passed that test


was (Author: lsharma3):
yes

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744039#comment-16744039
 ] 

Lucky Sharma commented on LUCENE-8640:
--

yes

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated LUCENE-8640:
-
Attachment: mypatch.patch

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated LUCENE-8640:
-
Attachment: (was: mypatch.patch)

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

Re: Admin UI - Collections Management

[Jan Høydahl]

I think perhaps this belongs in the Collections -> coll_name -> Shards -> 
Replicas screen, where you can already delete and add a replica to a shard. 
That screen has a dropdown to select what node the new replica should be 
created on. So if you add a MOVE button next to the DELETE button, and show a 
dropdown with list of nodes, then you should have enough info to construct a 
MOVEREPLICA cmd.

I know that there is a huge overlap between the Collections and Cloud menus, so 
at some point a redesign is needed. Of course we could also add a 
context-sensitive popup menu with a MOVE button to the Cloud->Nodes screen 
where we list replicas...

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 15. jan. 2019 kl. 19:33 skrev Erick Erickson :
> 
> bq: Thoughts?
> 
> I don't do UI work so I just sit back and cheer ;)
> 
> Then I do, of course have thoughts...
> 
> Mostly, when you're thinking about this think about scale. For
> instance, the graph view is nice for a few nodes/collections. 1,000
> collections (yes, some people scale that way) with even 2x2
> collections is almost, but not quite totally, unusable. Maybe
> copy/paste or cut/paste? Or some way to restrict what's shown to the
> relevant nodes? More generally how to navigate around when there are
> zillions of replicas
> 
> Best,
> Erick
> 
> 
> On Tue, Jan 15, 2019 at 8:39 AM Branham, Jeremy (Experis)
>  wrote:
>> 
>> Thanks Erick –
>> I used the MOVEREPLICA command to shuffle them around, but like you 
>> mentioned, I had to look up nodes names and such.
>> I’m on the same page; drag-n-drop replica organization.
>> 
>> I was looking at the newest section in the admin-ui  “Cloud > Nodes”
>> Maybe we could add the drag-n-drop functionality to that last column, where 
>> the replicas are listed by node. I think we have all the data required in 
>> that view to construct a MOVEREPLICA command.
>> 
>> Thoughts?
>> 
>> 
>> Jeremy Branham
>> jb...@allstate.com
>> 
>> On 1/14/19, 4:55 PM, "Erick Erickson"  wrote:
>> 
>>bq. The collections API allowed me to move the replicas around by
>>investigating the core names and locations,
>> 
>>What about the MOVEREPLICA command?
>>
>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_6-5F6_collections-2Dapi.html=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=037mwmgIodjUnHNITRxTA5FJu2QXK_AYmNgd-oCyfxU=
>> 
>>Although that still requires you to know things like node names and
>>replica names. It'd be way cool to have
>>some kind of drag-n-drop or wizard-driven capability, perhaps built on
>>top of that. In addition to Jan's
>>suggestions.
>> 
>>Best,
>>Erick
>> 
>>On Mon, Jan 14, 2019 at 2:37 PM Jan Høydahl  wrote:
>>> 
>>> Hi and thanks for offering to help.
>>> 
>>> If you are not familiar with the new Autoscaling framework, I'd start by 
>>> exploring it, since it aims at solving replica placement without explicit 
>>> commands.
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__lucene.apache.org_solr_guide_7-5F6_solrcloud-2Dautoscaling.html=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=SRn57bErK14_HIBo5zPxDGt8KgkslZhmB2zNNCd2jXc=
>>> 
>>> Next, I'd try to find an open JIRA issue to solve, perhaps something 
>>> related to Collections API and/or Admin UI. Pick something very simple, 
>>> just to get started with the procedure of contributing, and then look at 
>>> e.g. 
>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D10209=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=XB6iMPC0x1pEBUc-i-g8TwI9Y9QuQHcT87S-0Qx3NV8=
>>>  which seems related in that it aims to expose collection api through UI
>>> 
>>> --
>>> Jan Høydahl, search solution architect
>>> Cominvent AS - 
>>> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.cominvent.com=DwIFaQ=gtIjdLs6LnStUpy9cTOW9w=0SwsmPELGv6GC1_5JSQ9T7ZPMLljrIkbF_2jBCrKXI0=H4NKUT5exNIKdsoam9tBFwkVy-6YdmME9JkEo0ISjnk=YUNZMwT6xSa-P3X04jHMB58-A-jn5E9PII77JqfQceo=
>>> 
>>> 14. jan. 2019 kl. 21:39 skrev Branham, Jeremy (Experis) 
>>> :
>>> 
>>> I recently split some shards, and the new replicas didn’t go to the nodes I 
>>> wanted them.
>>> The collections API allowed me to move the replicas around by investigating 
>>> the core names and locations, then constructing the correct urls to execute 
>>> the moves.
>>> This worked, but it would have been faster if the admin UI supported such 
>>> operations.
>>> 
>>> Is this something I could contribute to? Maybe a PR in GitHub?
>>> I’ve been a solr user for quite a while and would like to start giving back 
>>> some.
>>> 
>>> Thanks!
>>> 
>>> Jeremy Branham
>>> jb...@allstate.com
>>> Allstate Insurance Company | UCV Technology Services | Information Services 

[jira] [Commented] (LUCENE-8634) LatLonShape: Query with the same polygon that is indexed might not match

[Adrien Grand (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743997#comment-16743997
 ] 

Adrien Grand commented on LUCENE-8634:
--

Thanks Ignacio. I understand how it makes things better but then I have two 
questions:
 - Are there concerns regarding the fact that a polygon might become invalid 
through encoding (and eg. cross itself).
 - Should we keep lines and polygons in the encoded space like boxes?

> LatLonShape: Query with the same polygon that is indexed might not match
> 
>
> Key: LUCENE-8634
> URL: https://issues.apache.org/jira/browse/LUCENE-8634
> Project: Lucene - Core
>  Issue Type: Bug
>  Components: modules/sandbox
>Affects Versions: 8.0, 7.7, master (9.0)
>Reporter: Ignacio Vera
>Priority: Major
> Attachments: LUCENE-8634.patch, LUCENE-8634.patch
>
>
> If a polygon with a degenerated dimension is indexed and then an intersect 
> query is performed with the same polygon, it might result in an empty result. 
> For example this polygon with degenerated longitude:
> POLYGON((1.401298464324817E-45 22.0, 1.401298464324817E-45 69.0, 
> 4.8202184588118395E-40 69.0, 4.8202184588118395E-40 22.0, 
> 1.401298464324817E-45 22.0))
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8640) validate delimiters when parsing date ranges

[Mikhail Khludnev (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743949#comment-16743949
 ] 

Mikhail Khludnev commented on LUCENE-8640:
--

does it pass Solr's DateRangeFieldTest ? 

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated LUCENE-8640:
-
Attachment: (was: mypatch.patch)

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated LUCENE-8640:
-
Attachment: mypatch.patch

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[JENKINS] Lucene-Solr-master-Linux (64bit/jdk-9.0.4) - Build # 23529 - Still unstable!

[Policeman Jenkins Server]

Build: https://jenkins.thetaphi.de/job/Lucene-Solr-master-Linux/23529/
Java: 64bit/jdk-9.0.4 -XX:-UseCompressedOops -XX:+UseSerialGC

4 tests failed.
FAILED:  org.apache.lucene.search.TermInSetQueryTest.testRamBytesUsed

Error Message:
expected:<237624.0> but was:<3456.0>

Stack Trace:
java.lang.AssertionError: expected:<237624.0> but was:<3456.0>
at 
__randomizedtesting.SeedInfo.seed([69A195B40323B4F5:9B0287F4C95CABA3]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:575)
at org.junit.Assert.assertEquals(Assert.java:700)
at 
org.apache.lucene.search.TermInSetQueryTest.testRamBytesUsed(TermInSetQueryTest.java:186)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at 
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at 
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at 
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at 
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at 
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at 
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at 
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at 
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:844)


FAILED:  org.apache.lucene.search.TermInSetQueryTest.testRamBytesUsed

Error Message:
expected:<237624.0> but was:<3456.0>

Stack Trace:
java.lang.AssertionError: expected:<237624.0> but was:<3456.0>
at 
__randomizedtesting.SeedInfo.seed([69A195B40323B4F5:9B0287F4C95CABA3]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:575)
at org.junit.Assert.assertEquals(Assert.java:700)
at 

[GitHub] MighTguY opened a new pull request #537: LUCENE-8640: added changes for the validation of valid dateString

[GitBox]

MighTguY opened a new pull request #537: LUCENE-8640: added changes for the 
validation of valid dateString
URL: https://github.com/apache/lucene-solr/pull/537
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743898#comment-16743898
 ] 

Lucky Sharma commented on LUCENE-8640:
--

Please review the patch for the same

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Lucky Sharma updated LUCENE-8640:
-
Attachment: mypatch.patch

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
> Attachments: mypatch.patch
>
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8633) Remove term weighting from interval scoring

[Jim Ferenczi (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743886#comment-16743886
 ] 

Jim Ferenczi commented on LUCENE-8633:
--

Thanks Alan, +1 the patch looks good.

> Remove term weighting from interval scoring
> ---
>
> Key: LUCENE-8633
> URL: https://issues.apache.org/jira/browse/LUCENE-8633
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8633.patch, LUCENE-8633.patch
>
>
> IntervalScorer currently uses the same scoring mechanism as SpanScorer, 
> summing the IDF of all possibly matching terms from its parent 
> IntervalsSource and using that in conjunction with a sloppy frequency to 
> produce a similarity-based score.  This doesn't really make sense, however, 
> as it means that terms that don't appear in a document can still contribute 
> to the score, and appears to make scores from interval queries comparable 
> with scores from term or phrase queries when they really aren't.
> I'd like to explore a different scoring mechanism for intervals, based purely 
> on sloppy frequency and ignoring term weighting.  This should make the scores 
> easier to reason about, as well as making them useful for things like 
> proximity boosting on boolean queries.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8644) Pass token types to QueryBuilder.newSynonymQuery()

[Jim Ferenczi (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743890#comment-16743890
 ] 

Jim Ferenczi commented on LUCENE-8644:
--

I like the idea but I wonder how it would work for multi-term synonyms ? 
Regarding the new signature, can we use an array of TermAndType rather than a 
Map, duplicate terms should be rare but still possible (they could also have 
different type).

> Pass token types to QueryBuilder.newSynonymQuery()
> --
>
> Key: LUCENE-8644
> URL: https://issues.apache.org/jira/browse/LUCENE-8644
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8644.patch
>
>
> SynonymGraphFilter distinguishes between input tokens and tokens that it 
> injects into the token stream by setting the TypeAttribute to SYNONYM.  
> QueryBuilder currently ignores this information, but it would be useful to 
> pass it through to the newSynonymQuery() method, as this would allow 
> queryparsers to apply differential boosts to synonyms and original terms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8644) Pass token types to QueryBuilder.newSynonymQuery()

[Alan Woodward (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743871#comment-16743871
 ] 

Alan Woodward commented on LUCENE-8644:
---

This patch changes the signature of newSynonymQuery from `(Term[)]` to 
`(Map)`, a map of terms to their types.  It also makes some 
changes to GraphTokenStreamFiniteStrings to keep track of the type attribute.  
It doesn't make any changes to query parser behaviour - this is purely to make 
the information available. 

> Pass token types to QueryBuilder.newSynonymQuery()
> --
>
> Key: LUCENE-8644
> URL: https://issues.apache.org/jira/browse/LUCENE-8644
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8644.patch
>
>
> SynonymGraphFilter distinguishes between input tokens and tokens that it 
> injects into the token stream by setting the TypeAttribute to SYNONYM.  
> QueryBuilder currently ignores this information, but it would be useful to 
> pass it through to the newSynonymQuery() method, as this would allow 
> queryparsers to apply differential boosts to synonyms and original terms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8643) TestLatLonLineShapeQueries.testRandomBig fails with suite timeout (simple text codec used)

[Ignacio Vera (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743869#comment-16743869
 ] 

Ignacio Vera commented on LUCENE-8643:
--

I agree it is over-iterating and that's the job of CI. And more over when one 
shape can generate several points :)

I will have a look the case of having big lines causing degrading performance 
and file an issue if needed.

 

> TestLatLonLineShapeQueries.testRandomBig fails with suite timeout (simple 
> text codec used)
> --
>
> Key: LUCENE-8643
> URL: https://issues.apache.org/jira/browse/LUCENE-8643
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> PR at: https://github.com/apache/lucene-solr/pull/536



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8640) validate delimiters when parsing date ranges

[Lucky Sharma (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8640?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743867#comment-16743867
 ] 

Lucky Sharma commented on LUCENE-8640:
--

[~mkhludnev] will this be count as a valid date -292275055-05-16T16:47:04.192 ?

> validate delimiters when parsing date ranges
> 
>
> Key: LUCENE-8640
> URL: https://issues.apache.org/jira/browse/LUCENE-8640
> Project: Lucene - Core
>  Issue Type: Improvement
>  Components: modules/spatial
>Reporter: Mikhail Khludnev
>Priority: Major
>
> {{DateRangePrefixTree.parseCalendar()}} should validate delimiters to rejects 
> dates like {{2000-11T13}} 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (LUCENE-8644) Pass token types to QueryBuilder.newSynonymQuery()

[Alan Woodward (JIRA)]

Alan Woodward created LUCENE-8644:
-

 Summary: Pass token types to QueryBuilder.newSynonymQuery()
 Key: LUCENE-8644
 URL: https://issues.apache.org/jira/browse/LUCENE-8644
 Project: Lucene - Core
  Issue Type: Improvement
Reporter: Alan Woodward
Assignee: Alan Woodward
 Attachments: LUCENE-8644.patch

SynonymGraphFilter distinguishes between input tokens and tokens that it 
injects into the token stream by setting the TypeAttribute to SYNONYM.  
QueryBuilder currently ignores this information, but it would be useful to pass 
it through to the newSynonymQuery() method, as this would allow queryparsers to 
apply differential boosts to synonyms and original terms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (LUCENE-8644) Pass token types to QueryBuilder.newSynonymQuery()

[Alan Woodward (JIRA)]

 [ 
https://issues.apache.org/jira/browse/LUCENE-8644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alan Woodward updated LUCENE-8644:
--
Attachment: LUCENE-8644.patch

> Pass token types to QueryBuilder.newSynonymQuery()
> --
>
> Key: LUCENE-8644
> URL: https://issues.apache.org/jira/browse/LUCENE-8644
> Project: Lucene - Core
>  Issue Type: Improvement
>Reporter: Alan Woodward
>Assignee: Alan Woodward
>Priority: Major
> Attachments: LUCENE-8644.patch
>
>
> SynonymGraphFilter distinguishes between input tokens and tokens that it 
> injects into the token stream by setting the TypeAttribute to SYNONYM.  
> QueryBuilder currently ignores this information, but it would be useful to 
> pass it through to the newSynonymQuery() method, as this would allow 
> queryparsers to apply differential boosts to synonyms and original terms.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (LUCENE-8643) TestLatLonLineShapeQueries.testRandomBig fails with suite timeout (simple text codec used)

[Dawid Weiss (JIRA)]

[ 
https://issues.apache.org/jira/browse/LUCENE-8643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743857#comment-16743857
 ] 

Dawid Weiss commented on LUCENE-8643:
-

I think it's just terribly slow if it hits a high number of shapes and a high 
iteration count. In line with recent issue me and Adrien discussed: I don't 
think it makes sense to beat those tests to death with internal iteration 
counts; should something be wrong, repeated CI builds will capture it. 

I decreased iteration counts and marked repeated tests as slow, excluding 
simple text codec as well. Please feel free to file another issue if there's 
really an other underlying problem you see, Ignacio (but I see these tests as 
terribly slow in pretty much any run).

> TestLatLonLineShapeQueries.testRandomBig fails with suite timeout (simple 
> text codec used)
> --
>
> Key: LUCENE-8643
> URL: https://issues.apache.org/jira/browse/LUCENE-8643
> Project: Lucene - Core
>  Issue Type: Bug
>Reporter: Dawid Weiss
>Assignee: Dawid Weiss
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> PR at: https://github.com/apache/lucene-solr/pull/536



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org