[jira] [Assigned] (SOLR-7323) Core Admin API looks for config sets in wrong directory
[
https://issues.apache.org/jira/browse/SOLR-7323?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley reassigned SOLR-7323:
--
Assignee: David Smiley
> Core Admin API looks for config sets in wrong directory
> ---
>
> Key: SOLR-7323
> URL: https://issues.apache.org/jira/browse/SOLR-7323
> Project: Solr
> Issue Type: Bug
> Components: Server
>Affects Versions: 5.0
>Reporter: Mark Haase
>Assignee: David Smiley
>Priority: Major
>
> *To Reproduce*
> Try to create a core using Core Admin API and a config set:
> {code}
> curl
> 'http://localhost:8983/solr/admin/cores?action=CREATE=new_core=basic_configs'
> {code}
> *Expected Outcome*
> Core is created in `/var/solr/data/new_core` using one of the config sets
> installed by the installer script in
> `/opt/solr/server/solr/configsets/basic_configs`.
> *Actual Outcome*
> {code}
>
>
> 400 name="QTime">9Error CREATEing
> SolrCore 'new_core': Unable to create core [new_core] Caused by: Could not
> load configuration from directory
> /var/solr/data/configsets/basic_configs400
>
> {code}
> Why is it looking for config sets in /var/solr/data? I don't know. If that's
> where configsets are supposed to be placed, then why does the installer put
> them somewhere else?
> There's no documented API to tell it to look for config sets anywhere else,
> either. It will always search inside /var/solr/data.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14494) Refactor BlockJoin to not use Filter.java
[
https://issues.apache.org/jira/browse/SOLR-14494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120059#comment-17120059
]
ASF subversion and git services commented on SOLR-14494:
Commit a6a02ac0e5116f4cf061fd3afeb930c3cc379f5a in lucene-solr's branch
refs/heads/master from David Smiley
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a6a02ac ]
SOLR-14494: Refactor BlockJoin to not use Filter (#1523)
Note: henceforth the perSegFilter cache will internally have values of type
BitSetProducer instead of Filter.
> Refactor BlockJoin to not use Filter.java
> -
>
> Key: SOLR-14494
> URL: https://issues.apache.org/jira/browse/SOLR-14494
> Project: Solr
> Issue Type: Sub-task
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The class
> {{org.apache.solr.search.join.BlockJoinParentQParser.BitDocIdSetFilterWrapper}}
> extends Filter, but Filter is going away eventually.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14494) Refactor BlockJoin to not use Filter.java
[
https://issues.apache.org/jira/browse/SOLR-14494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Smiley resolved SOLR-14494.
-
Fix Version/s: master (9.0)
Resolution: Fixed
> Refactor BlockJoin to not use Filter.java
> -
>
> Key: SOLR-14494
> URL: https://issues.apache.org/jira/browse/SOLR-14494
> Project: Solr
> Issue Type: Sub-task
>Reporter: David Smiley
>Assignee: David Smiley
>Priority: Major
> Fix For: master (9.0)
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> The class
> {{org.apache.solr.search.join.BlockJoinParentQParser.BitDocIdSetFilterWrapper}}
> extends Filter, but Filter is going away eventually.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley merged pull request #1523: SOLR-14494: Refactor BlockJoin to not use Filter
dsmiley merged pull request #1523: URL: https://github.com/apache/lucene-solr/pull/1523 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9387) Remove RAM accounting from LeafReader
[
https://issues.apache.org/jira/browse/LUCENE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120004#comment-17120004
]
Michael McCandless commented on LUCENE-9387:
{quote}I'm not sure we need to tie this with the move of live docs off-heap:
live docs are ignored from CodecReader#ramBytesUsed anyway.
{quote}
Ahh OK, that is weird :) But, yes, let's not couple the two issues then.
> Remove RAM accounting from LeafReader
> -
>
> Key: LUCENE-9387
> URL: https://issues.apache.org/jira/browse/LUCENE-9387
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
>
> Context for this issue can be found at
> https://lists.apache.org/thread.html/r06b6a63d8689778bbc2736ec7e4e39bf89ae6973c19f2ec6247690fd%40%3Cdev.lucene.apache.org%3E.
> RAM accounting made sense when readers used lots of memory. E.g. when norms
> were on heap, we could return memory usage of the norms array and memory
> estimates would be very close to actual memory usage.
> However nowadays, readers consume very little memory, so RAM accounting has
> become less valuable. Furthermore providing good estimates has become
> incredibly complex as we can no longer focus on a couple main contributors to
> memory usage, but would need to start considering things that we historically
> ignored, such as field infos, segment infos, NIOFS buffers, etc.
> Let's remove RAM accounting from LeafReader?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9388) Move live docs to disk?
[
https://issues.apache.org/jira/browse/LUCENE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120001#comment-17120001
]
Michael McCandless commented on LUCENE-9388:
Memory mapped IO is likely plenty fast, but there may indeed be some overhead
versus the {{long[]}} that backs our typical live docs implementations?
Since live docs are mostly accessed forward only iterator style (like doc
values), maybe switching the random access {{BitSet}} we use today to an
iterator over bits is the first step? We might also want to invert it
(again!!), so the iterator's {{.next()}} advances to the next deleted doc,
since for most indices more documents are live than not.
We should just try it and see :)
> Move live docs to disk?
> ---
>
> Key: LUCENE-9388
> URL: https://issues.apache.org/jira/browse/LUCENE-9388
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
>
> Live docs are the last file format whose memory usage is a function of
> maxDoc, let's look into moving them to disk?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9388) Move live docs to disk?
[ https://issues.apache.org/jira/browse/LUCENE-9388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119956#comment-17119956 ] David Smiley commented on LUCENE-9388: -- This idea surprises me. Doesn't this need to be an efficient BitSet? Is accessing memory-mapped data approximately zero relative to on-heap? > Move live docs to disk? > --- > > Key: LUCENE-9388 > URL: https://issues.apache.org/jira/browse/LUCENE-9388 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > > Live docs are the last file format whose memory usage is a function of > maxDoc, let's look into moving them to disk? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119944#comment-17119944 ] ASF subversion and git services commented on LUCENE-9359: - Commit 22cb4d4d95ee854aabe5e2a52639f43090fef656 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=22cb4d4 ] LUCENE-9359: Address test failures when the codec version gets modified. > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119943#comment-17119943 ] ASF subversion and git services commented on LUCENE-9359: - Commit ec1757b7c21bf023b31853aabe2f4f4d2a46c18e in lucene-solr's branch refs/heads/branch_8x from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=ec1757b ] LUCENE-9359: Address test failures when the codec version gets modified. > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9301) Gradle: Jar MANIFEST incomplete
[
https://issues.apache.org/jira/browse/LUCENE-9301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119876#comment-17119876
]
David Smiley commented on LUCENE-9301:
--
+1 to the patch
> Gradle: Jar MANIFEST incomplete
> ---
>
> Key: LUCENE-9301
> URL: https://issues.apache.org/jira/browse/LUCENE-9301
> Project: Lucene - Core
> Issue Type: Sub-task
> Components: general/build
>Affects Versions: master (9.0)
>Reporter: Jan Høydahl
>Assignee: Dawid Weiss
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: LUCENE-9301.patch
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After building with gradle, the MANIFEST.MF file for e.g. solr-core.jar
> containst
> {noformat}
> Manifest-Version: 1.0
> {noformat}
> While when building with ant, it says
> {noformat}
> Manifest-Version: 1.0
> Ant-Version: Apache Ant 1.10.7
> Created-By: 11.0.6+10 (AdoptOpenJDK)
> Extension-Name: org.apache.solr
> Specification-Title: Apache Solr Search Server: solr-core
> Specification-Version: 9.0.0
> Specification-Vendor: The Apache Software Foundation
> Implementation-Title: org.apache.solr
> Implementation-Version: 9.0.0-SNAPSHOT 9b5542ad55da601e0bdfda96bad8c2c
> cabbbc397 - janhoy - 2020-04-01 16:24:09
> Implementation-Vendor: The Apache Software Foundation
> X-Compile-Source-JDK: 11
> X-Compile-Target-JDK: 11
> {noformat}
> In addition, with ant, the META-INF folder also contains LICENSE.txt and
> NOTICE.txt files.
> There is a macro {{build-manifest}} in common-build.xml that seems to build
> the manifest.
> The effect of this is e.g. that spec an implementation versions do not show
> in Solr Admin UI
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9387) Remove RAM accounting from LeafReader
[ https://issues.apache.org/jira/browse/LUCENE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119868#comment-17119868 ] Adrien Grand commented on LUCENE-9387: -- I'm not sure we need to tie this with the move of live docs off-heap: live docs are ignored from CodecReader#ramBytesUsed anyway. > Remove RAM accounting from LeafReader > - > > Key: LUCENE-9387 > URL: https://issues.apache.org/jira/browse/LUCENE-9387 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > > Context for this issue can be found at > https://lists.apache.org/thread.html/r06b6a63d8689778bbc2736ec7e4e39bf89ae6973c19f2ec6247690fd%40%3Cdev.lucene.apache.org%3E. > RAM accounting made sense when readers used lots of memory. E.g. when norms > were on heap, we could return memory usage of the norms array and memory > estimates would be very close to actual memory usage. > However nowadays, readers consume very little memory, so RAM accounting has > become less valuable. Furthermore providing good estimates has become > incredibly complex as we can no longer focus on a couple main contributors to > memory usage, but would need to start considering things that we historically > ignored, such as field infos, segment infos, NIOFS buffers, etc. > Let's remove RAM accounting from LeafReader? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13749) Implement support for joining across collections with multiple shards ( XCJF )
[
https://issues.apache.org/jira/browse/SOLR-13749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119856#comment-17119856
]
Dan Fox commented on SOLR-13749:
I updated the PR to use {{method=crossCollection}} instead of
{{method=ccjoin}}, and to remove the CrossCollectionJoinQParserPlugin class
which is no longer necessary.
> Implement support for joining across collections with multiple shards ( XCJF )
> --
>
> Key: SOLR-13749
> URL: https://issues.apache.org/jira/browse/SOLR-13749
> Project: Solr
> Issue Type: New Feature
>Reporter: Kevin Watters
>Assignee: Gus Heck
>Priority: Blocker
> Fix For: 8.6
>
> Attachments: 2020-03 Smiley with ASF hat.jpeg
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> This ticket includes 2 query parsers.
> The first one is the "Cross collection join filter" (XCJF) parser. This is
> the "Cross-collection join filter" query parser. It can do a call out to a
> remote collection to get a set of join keys to be used as a filter against
> the local collection.
> The second one is the Hash Range query parser that you can specify a field
> name and a hash range, the result is that only the documents that would have
> hashed to that range will be returned.
> This query parser will do an intersection based on join keys between 2
> collections.
> The local collection is the collection that you are searching against.
> The remote collection is the collection that contains the join keys that you
> want to use as a filter.
> Each shard participating in the distributed request will execute a query
> against the remote collection. If the local collection is setup with the
> compositeId router to be routed on the join key field, a hash range query is
> applied to the remote collection query to only match the documents that
> contain a potential match for the documents that are in the local shard/core.
>
>
> Here's some vocab to help with the descriptions of the various parameters.
> ||Term||Description||
> |Local Collection|This is the main collection that is being queried.|
> |Remote Collection|This is the collection that the XCJFQuery will query to
> resolve the join keys.|
> |XCJFQuery|The lucene query that executes a search to get back a set of join
> keys from a remote collection|
> |HashRangeQuery|The lucene query that matches only the documents whose hash
> code on a field falls within a specified range.|
>
>
> ||Param ||Required ||Description||
> |collection|Required|The name of the external Solr collection to be queried
> to retrieve the set of join key values ( required )|
> |zkHost|Optional|The connection string to be used to connect to Zookeeper.
> zkHost and solrUrl are both optional parameters, and at most one of them
> should be specified.
> If neither of zkHost or solrUrl are specified, the local Zookeeper cluster
> will be used. ( optional )|
> |solrUrl|Optional|The URL of the external Solr node to be queried ( optional
> )|
> |from|Required|The join key field name in the external collection ( required
> )|
> |to|Required|The join key field name in the local collection|
> |v|See Note|The query to be executed against the external Solr collection to
> retrieve the set of join key values.
> Note: The original query can be passed at the end of the string or as the
> "v" parameter.
> It's recommended to use query parameter substitution with the "v" parameter
> to ensure no issues arise with the default query parsers.|
> |routed| |true / false. If true, the XCJF query will use each shard's hash
> range to determine the set of join keys to retrieve for that shard.
> This parameter improves the performance of the cross-collection join, but
> it depends on the local collection being routed by the toField. If this
> parameter is not specified,
> the XCJF query will try to determine the correct value automatically.|
> |ttl| |The length of time that an XCJF query in the cache will be considered
> valid, in seconds. Defaults to 3600 (one hour).
> The XCJF query will not be aware of changes to the remote collection, so
> if the remote collection is updated, cached XCJF queries may give inaccurate
> results.
> After the ttl period has expired, the XCJF query will re-execute the join
> against the remote collection.|
> |_All others_| |Any normal Solr parameter can also be specified as a local
> param.|
>
> Example Solr Config.xml changes:
>
> {{<}}{{cache}} {{name}}{{=}}{{"hash_vin"}}
> {{ }}{{class}}{{=}}{{"solr.LRUCache"}}
> {{ }}{{size}}{{=}}{{"128"}}
> {{ }}{{initialSize}}{{=}}{{"0"}}
> {{ }}{{regenerator}}{{=}}{{"solr.NoOpRegenerator"}}{{/>}}
>
> {{<}}{{queryParser}} {{name}}{{=}}{{"xcjf"}}
>
[jira] [Commented] (LUCENE-9387) Remove RAM accounting from LeafReader
[
https://issues.apache.org/jira/browse/LUCENE-9387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119826#comment-17119826
]
Michael McCandless commented on LUCENE-9387:
+1 to remove {{Accountable}} from Lucene's reader hierarchy, maybe after we
move live docs off heap?
> Remove RAM accounting from LeafReader
> -
>
> Key: LUCENE-9387
> URL: https://issues.apache.org/jira/browse/LUCENE-9387
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
>
> Context for this issue can be found at
> https://lists.apache.org/thread.html/r06b6a63d8689778bbc2736ec7e4e39bf89ae6973c19f2ec6247690fd%40%3Cdev.lucene.apache.org%3E.
> RAM accounting made sense when readers used lots of memory. E.g. when norms
> were on heap, we could return memory usage of the norms array and memory
> estimates would be very close to actual memory usage.
> However nowadays, readers consume very little memory, so RAM accounting has
> become less valuable. Furthermore providing good estimates has become
> incredibly complex as we can no longer focus on a couple main contributors to
> memory usage, but would need to start considering things that we historically
> ignored, such as field infos, segment infos, NIOFS buffers, etc.
> Let's remove RAM accounting from LeafReader?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9365) Fuzzy query has a false negative when prefix length == search term length
[
https://issues.apache.org/jira/browse/LUCENE-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119707#comment-17119707
]
Mike Drob commented on LUCENE-9365:
---
[~mikemccand] - tested it, seems to fix the corner case, I'm not sure if there
is some other undesirable behavior that pops up from removing this though.
Maybe it's better to keep the {{maxEdits == 0}} part of the check and still
allow SingleTermEnum to exist here. Put up a PR with [~mharwood]'s original
test and the new change.
> Fuzzy query has a false negative when prefix length == search term length
> --
>
> Key: LUCENE-9365
> URL: https://issues.apache.org/jira/browse/LUCENE-9365
> Project: Lucene - Core
> Issue Type: Bug
> Components: core/query/scoring
>Reporter: Mark Harwood
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> When using FuzzyQuery the search string `bba` does not match doc value `bbab`
> with an edit distance of 1 and prefix length of 3.
> In FuzzyQuery an automaton is created for the "suffix" part of the search
> string which in this case is an empty string.
> In this scenario maybe the FuzzyQuery should rewrite to a WildcardQuery of
> the following form :
> {code:java}
> searchString + "?"
> {code}
> .. where there's an appropriate number of ? characters according to the edit
> distance.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on pull request #1543: LUCENE-9378: Disable compression on binary values whose length is less than 32.
msokolov commented on pull request #1543: URL: https://github.com/apache/lucene-solr/pull/1543#issuecomment-636035462 I guess I'm concerned that there can still be perf degradation for longer highly-compressible fields. Still perhaps we can extend this approach by providing a per-field configuration if that becomes needed, so let's merge and make some progress! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] madrob opened a new pull request #1545: LUCENE-9365 FuzzyQuery false negative
madrob opened a new pull request #1545: URL: https://github.com/apache/lucene-solr/pull/1545 when prefix length == search term length Co-Authored-By: markharwood This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13482) solr-test-framework - junit jupiter support
[ https://issues.apache.org/jira/browse/SOLR-13482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bernd Wahlen updated SOLR-13482: Affects Version/s: (was: 7.6) (was: 8.0) 8.5.1 > solr-test-framework - junit jupiter support > --- > > Key: SOLR-13482 > URL: https://issues.apache.org/jira/browse/SOLR-13482 > Project: Solr > Issue Type: Wish > Components: clients - java, Tests >Affects Versions: 8.5.1 >Reporter: Bernd Wahlen >Priority: Minor > > As far as i know solr-test-framework supports only junit4 style test via > SolrTestCaseJ4 and does not provide junit5 extension. I know that i can still > run tests with junit vintage engine. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14426) forbidden api error during precommit DateMathFunction
[
https://issues.apache.org/jira/browse/SOLR-14426?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119697#comment-17119697
]
Mike Drob commented on SOLR-14426:
--
[~erickerickson] - what's the state here? I'm happy to follow your lead and
move classes to top level, but I'm not clear on what the consensus is.
> forbidden api error during precommit DateMathFunction
> -
>
> Key: SOLR-14426
> URL: https://issues.apache.org/jira/browse/SOLR-14426
> Project: Solr
> Issue Type: Bug
> Components: Build
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Major
> Fix For: master (9.0)
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> When running `./gradlew precommit` I'll occasionally see
> {code}
> * What went wrong:
> Execution failed for task ':solr:contrib:analytics:forbiddenApisMain'.
> > de.thetaphi.forbiddenapis.ForbiddenApiException: Check for forbidden API
> > calls failed while scanning class
> > 'org.apache.solr.analytics.function.mapping.DateMathFunction'
> > (DateMathFunction.java): java.lang.ClassNotFoundException:
> > org.apache.solr.analytics.function.mapping.DateMathValueFunction (while
> > looking up details about referenced class
> > 'org.apache.solr.analytics.function.mapping.DateMathValueFunction')
> {code}
> `./gradlew clean` fixes this, but I don't understand what or why this
> happens. Feels like a gradle issue?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] msokolov commented on pull request #1543: LUCENE-9378: Disable compression on binary values whose length is less than 32.
msokolov commented on pull request #1543: URL: https://github.com/apache/lucene-solr/pull/1543#issuecomment-636020925 Thanks Adrien - speedup looks good for the BDVSort case, but I wonder if it has really recovered to the status quo ante. Were you able to run a before/after with 8.4? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119664#comment-17119664 ] ASF subversion and git services commented on LUCENE-9359: - Commit a848525855b47cd911414cb2e535bce276e200ad in lucene-solr's branch refs/heads/branch_8x from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=a848525 ] LUCENE-9359: Always call checkFooter in SegmentInfos#readCommit. (#1483) > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119663#comment-17119663 ] ASF subversion and git services commented on LUCENE-9359: - Commit f908f2cd482a5815fa29344987ee801420aa8de0 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=f908f2c ] LUCENE-9359: Always call checkFooter in SegmentInfos#readCommit. (#1483) > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119617#comment-17119617 ] ASF subversion and git services commented on LUCENE-9359: - Commit fe07d9dd0e343754a772e90dce33a17f5cbb19d3 in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=fe07d9d ] Revert "LUCENE-9359: Always call checkFooter in SegmentInfos#readCommit. (#1483)" This reverts commit bfb6bf9c9aafc778a88000e87f082d82dba9872c. > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-8673) o.a.s.search.facet classes not public/extendable
[
https://issues.apache.org/jira/browse/SOLR-8673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119612#comment-17119612
]
Tim Owen commented on SOLR-8673:
Yes this is annoying, I was mid-way through writing a custom aggregate today
until I hit this problem.
Looks like it might be fixed very recently in the master branch, though.. work
done in SOLR-14482 has extracted FacetContext to a top-level class and made it
public at the same time.
> o.a.s.search.facet classes not public/extendable
>
>
> Key: SOLR-8673
> URL: https://issues.apache.org/jira/browse/SOLR-8673
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 5.4.1
>Reporter: Markus Jelsma
>Priority: Major
> Fix For: 6.2, 7.0
>
>
> It is not easy to create a custom JSON facet function. A simple function
> based on AvgAgg quickly results in the following compilation failures:
> {code}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.3:compile (default-compile)
> on project openindex-solr: Compilation failure: Compilation failure:
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[22,36]
> org.apache.solr.search.facet.FacetContext is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[23,36]
> org.apache.solr.search.facet.FacetDoubleMerger is not public in
> org.apache.solr.search.facet; cannot be accessed from outside package
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[40,32]
> cannot find symbol
> [ERROR] symbol: class FacetContext
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[49,39]
> cannot find symbol
> [ERROR] symbol: class FacetDoubleMerger
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[54,43]
> cannot find symbol
> [ERROR] symbol: class Context
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg.Merger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[41,16]
> cannot find symbol
> [ERROR] symbol: class AvgSlotAcc
> [ERROR] location: class i.o.s.search.facet.CustomAvgAgg
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[46,12]
> incompatible types: i.o.s.search.facet.CustomAvgAgg.Merger cannot be
> converted to org.apache.solr.search.facet.FacetMerger
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[53,5]
> method does not override or implement a method from a supertype
> [ERROR]
> /home/markus/projects/openindex/solr/trunk/src/main/java/i.o.s.search/facet/CustomAvgAgg.java:[60,5]
> method does not override or implement a method from a supertype
> {code}
> It seems lots of classes are tucked away in FacetModule, which we can't reach
> from outside.
> Originates from this thread:
> http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201602.mbox/%3ccab_8yd9ldbg_0zxm_h1igkfm6bqeypd5ilyy7tty8cztscv...@mail.gmail.com%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] markharwood commented on a change in pull request #1543: LUCENE-9378: Disable compression on binary values whose length is less than 32.
markharwood commented on a change in pull request #1543:
URL: https://github.com/apache/lucene-solr/pull/1543#discussion_r432467975
##
File path:
lucene/core/src/java/org/apache/lucene/codecs/lucene80/Lucene80DocValuesConsumer.java
##
@@ -429,7 +429,18 @@ private void flushData() throws IOException {
}
}
maxUncompressedBlockLength = Math.max(maxUncompressedBlockLength,
uncompressedBlockLength);
-LZ4.compress(block, 0, uncompressedBlockLength, data, ht);
+
+// Compression proved to hurt latency in some cases, so we're only
+// enabling it on long inputs for now. Can we reduce the compression
+// overhead and enable compression again, e.g. by building shared
+// dictionaries that allow decompressing one value at once instead of
+// forcing 32 values to be decompressed even when you only need one?
+if (uncompressedBlockLength >= 32 * numDocsInCurrentBlock) {
+ LZ4.compress(block, 0, uncompressedBlockLength, data, highCompHt);
Review comment:
Maybe create a constant for this 32 ("MIN_COMPRESSABLE_FIELD_LENGTH" ?)
otherwise it might get confused with the 32 we happen to use for max num of
values in a block.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand resolved LUCENE-9359. -- Fix Version/s: 8.6 master (9.0) Resolution: Fixed > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: master (9.0), 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrien Grand updated LUCENE-9359: - Fix Version/s: (was: master (9.0)) > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Fix For: 8.6 > > Time Spent: 20m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9359) SegmentInfos.readCommit should verify checksums in case of error
[ https://issues.apache.org/jira/browse/LUCENE-9359?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119570#comment-17119570 ] ASF subversion and git services commented on LUCENE-9359: - Commit bfb6bf9c9aafc778a88000e87f082d82dba9872c in lucene-solr's branch refs/heads/master from Adrien Grand [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=bfb6bf9 ] LUCENE-9359: Always call checkFooter in SegmentInfos#readCommit. (#1483) > SegmentInfos.readCommit should verify checksums in case of error > > > Key: LUCENE-9359 > URL: https://issues.apache.org/jira/browse/LUCENE-9359 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Adrien Grand >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > SegmentInfos.readCommit only calls checkFooter if reading the commit > succeeded. We should also call it in case of errors in order to be able to > distinguish hardware errors from Lucene bugs. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz merged pull request #1483: LUCENE-9359: Always call checkFooter in SegmentInfos#readCommit.
jpountz merged pull request #1483: URL: https://github.com/apache/lucene-solr/pull/1483 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy commented on pull request #1544: ref_guide - metrics reporting - small typo
janhoy commented on pull request #1544: URL: https://github.com/apache/lucene-solr/pull/1544#issuecomment-635953769 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] janhoy merged pull request #1544: ref_guide - metrics reporting - small typo
janhoy merged pull request #1544: URL: https://github.com/apache/lucene-solr/pull/1544 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (LUCENE-9389) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/LUCENE-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson reassigned LUCENE-9389:
--
Assignee: Erick Erickson
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: LUCENE-9389
> URL: https://issues.apache.org/jira/browse/LUCENE-9389
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Andras Salamon
>Assignee: Erick Erickson
>Priority: Minor
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14523) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/SOLR-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson updated SOLR-14523:
--
Attachment: validation.patch
Status: Open (was: Open)
[~asalamon74] Are you on the Apache Slack channel? Or maybe we can connect on
Google? Something interactive if the short form below doesn't work. I've
outlined each step I use, although you are probably familiar with most of them
already.
- Apply the "validation.patch" I've attached.
- I often start with "git clean -dxf" if I've used Ant because sometimes mixing
Ant and Gradle can confuse things.
- "gradlew idea" will create the project that you can open up in IntelliJ, go
ahead and open it.
- IntelliJ should ask you if you want it to import the Gradle project, say yes.
- In my setup, over on the right side there's both an "ant" and "gradle" tab.
Usually the gradle window opens automatically if I import the Gradle project,
but if not It's easy to miss, it's on the _extreme_ right. Find it and click it.
- Expand the project, then Tasks, then verification, then double-click
"validateLogCalls". You can execute this either at the root, or the individual
lucene or solr nodes.
what you should see pretty soon is a window open up with any violations
reported and clicking on the link in that window should take you to the source
line.
Let me know if any of that doesn't work. I'm using the Community Edition of
IntelliJ, 2020.1.1
And here's one case where I don't follow the patch naming convention. When
these are all fixed up, we can include the gradle change in the final patch.
Oh, and if you find any cases that the valiation task misses, let me know
and/or just fix the code if you're comfortable with Groovy in
"./gradle/validation/validate-log-calls.gradle"
Finally, "gradlew helpValidateLogCalls" will print out a lot of text about
this, if you'd like to add anything for getMessage and/or GetCause, just edit
help/validateLogCalls.txt
Thanks!
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: SOLR-14523
> URL: https://issues.apache.org/jira/browse/SOLR-14523
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andras Salamon
>Assignee: Erick Erickson
>Priority: Minor
> Attachments: validation.patch
>
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14280) SolrConfig logging not helpful
[
https://issues.apache.org/jira/browse/SOLR-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119532#comment-17119532
]
Erick Erickson commented on SOLR-14280:
---
Sure. I'll leave a comment over at SOLR-14523 momentarily.
> SolrConfig logging not helpful
> --
>
> Key: SOLR-14280
> URL: https://issues.apache.org/jira/browse/SOLR-14280
> Project: Solr
> Issue Type: Bug
>Reporter: Andras Salamon
>Assignee: Jason Gerlowski
>Priority: Minor
> Fix For: master (9.0), 8.6
>
> Attachments: SOLR-14280-01.patch, SOLR-14280-02.patch, getmessages.txt
>
>
> SolrConfig prints out a warning message if it's not able to add files to the
> classpath, but this message is not too helpful:
> {noformat}
> o.a.s.c.SolrConfig Couldn't add files from
> /opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.1850855/lib/solr/dist filtered
> by solr-langid-\d.*\.jar to classpath:
> /opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.1850855/lib/solr/
> dist {noformat}
> The reason should be at the end of the log message but it's just repeats the
> problematic file name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-14523) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/SOLR-14523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erick Erickson reassigned SOLR-14523:
-
Assignee: Erick Erickson
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: SOLR-14523
> URL: https://issues.apache.org/jira/browse/SOLR-14523
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Andras Salamon
>Assignee: Erick Erickson
>Priority: Minor
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] soleuu opened a new pull request #1544: ref_guide - metrics reporting - small typo
soleuu opened a new pull request #1544: URL: https://github.com/apache/lucene-solr/pull/1544 just a small typo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] markharwood commented on pull request #1541: RegExp - add case insensitive matching option
markharwood commented on pull request #1541: URL: https://github.com/apache/lucene-solr/pull/1541#issuecomment-635913889 >maybe we should make the constructor take an additional boolean instead of expecting users to configure case insensitivity via a syntax flag? Does it change things if we consider Java's case insensitivity is also [a bit mask flag passed to the constructor](https://docs.oracle.com/javase/7/docs/api/java/util/regex/Pattern.html#compile(java.lang.String,%20int))? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14442) bin/solr to attempt jstack before killing hung Solr instance
[
https://issues.apache.org/jira/browse/SOLR-14442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119474#comment-17119474
]
Christine Poerschke commented on SOLR-14442:
Thanks [~mkhl]! I won't get to this this week, sorry, i.e. please feel free to
go ahead and commit, thank you.
> bin/solr to attempt jstack before killing hung Solr instance
>
>
> Key: SOLR-14442
> URL: https://issues.apache.org/jira/browse/SOLR-14442
> Project: Solr
> Issue Type: Improvement
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Attachments: SOLR-14442.patch, SOLR-14442.patch, SOLR-14442.patch,
> screenshot-1.png
>
>
> If a Solr instance did not respond to the 'stop' command in a timely manner
> then the {{bin/solr}} script will attempt to forcefully kill it:
> [https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.5.1/solr/bin/solr#L859]
> Gathering of information (e.g. a jstack of the java process) before the kill
> command may be helpful in determining why the instance did not stop as
> expected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz commented on pull request #1543: LUCENE-9378: Disable compression on binary values whose length is less than 32.
jpountz commented on pull request #1543: URL: https://github.com/apache/lucene-solr/pull/1543#issuecomment-635887870 Here are results on wikimedium10m ``` TaskQPS baseline StdDev QPS patch StdDev Pct diff BrowseDayOfYearSSDVFacets 15.32 (8.5%) 14.80 (12.6%) -3.4% ( -22% - 19%) BrowseMonthSSDVFacets 16.73 (13.8%) 16.45 (14.8%) -1.7% ( -26% - 31%) HighSloppyPhrase 24.95 (8.0%) 24.66 (8.7%) -1.2% ( -16% - 16%) MedSloppyPhrase 171.40 (6.0%) 169.44 (6.4%) -1.1% ( -12% - 12%) OrNotHighMed 1305.52 (4.4%) 1296.78 (3.7%) -0.7% ( -8% -7%) LowSloppyPhrase 202.88 (4.6%) 201.78 (4.6%) -0.5% ( -9% -9%) HighTermMonthSort 90.33 (10.5%) 89.88 (10.8%) -0.5% ( -19% - 23%) AndHighLow 1221.67 (4.4%) 1216.14 (3.2%) -0.5% ( -7% -7%) MedTerm 2535.67 (3.4%) 2525.61 (3.0%) -0.4% ( -6% -6%) HighSpanNear 22.82 (4.4%) 22.76 (3.0%) -0.3% ( -7% -7%) HighTerm 2484.93 (3.5%) 2482.72 (2.8%) -0.1% ( -6% -6%) LowSpanNear 54.72 (2.9%) 54.67 (2.6%) -0.1% ( -5% -5%) OrHighNotHigh 1050.37 (2.8%) 1049.92 (3.2%) -0.0% ( -5% -6%) OrHighNotLow 1605.11 (3.6%) 1604.69 (3.4%) -0.0% ( -6% -7%) OrHighLow 803.31 (3.7%) 803.36 (3.9%) 0.0% ( -7% -7%) OrHighNotMed 1394.34 (3.2%) 1395.18 (3.3%) 0.1% ( -6% -6%) HighPhrase 250.77 (2.9%) 251.05 (3.0%) 0.1% ( -5% -6%) Prefix3 657.61 (9.6%) 658.52 (10.2%) 0.1% ( -17% - 21%) OrNotHighLow 1094.18 (3.9%) 1096.03 (3.2%) 0.2% ( -6% -7%) MedPhrase 167.34 (2.8%) 167.64 (2.6%) 0.2% ( -5% -5%) AndHighMed 127.35 (3.6%) 127.59 (2.8%) 0.2% ( -6% -6%) OrNotHighHigh 1193.60 (3.4%) 1195.83 (2.8%) 0.2% ( -5% -6%) OrHighMed 123.41 (4.0%) 123.74 (3.0%) 0.3% ( -6% -7%) Respell 305.95 (3.6%) 306.79 (4.4%) 0.3% ( -7% -8%) Wildcard 250.69 (7.3%) 251.41 (7.5%) 0.3% ( -13% - 16%) OrHighHigh 23.79 (4.3%) 23.88 (3.3%) 0.4% ( -6% -8%) PKLookup 257.43 (6.5%) 258.43 (5.0%) 0.4% ( -10% - 12%) MedSpanNear 15.20 (3.1%) 15.28 (2.6%) 0.5% ( -5% -6%) LowPhrase 947.49 (2.8%) 952.85 (2.3%) 0.6% ( -4% -5%) HighTermDayOfYearSort 88.51 (6.4%) 89.20 (5.5%) 0.8% ( -10% - 13%) AndHighHigh 118.29 (3.1%) 119.51 (2.9%) 1.0% ( -4% -7%) HighIntervalsOrdered 44.70 (4.0%) 45.35 (4.6%) 1.5% ( -6% - 10%) LowTerm 2756.66 (5.8%) 2806.48 (4.0%) 1.8% ( -7% - 12%) Fuzzy2 135.58 (8.8%) 139.92 (6.4%) 3.2% ( -11% - 20%) Fuzzy1 225.49 (9.3%) 234.07 (9.7%) 3.8% ( -13% - 25%) IntNRQ 100.54 (49.9%) 105.58 (49.7%) 5.0% ( -63% - 208%) BrowseDayOfYearTaxoFacets5.64 (8.7%)6.54 (11.0%) 15.9% ( -3% - 38%) BrowseDateTaxoFacets5.64 (8.6%)6.56 (10.9%) 16.3% ( -2% - 39%) BrowseMonthTaxoFacets6.67 (7.3%)7.84 (8.8%) 17.5% ( 1% - 36%) HighTermTitleBDVSort6.16 (6.1%) 14.02 (28.0%) 127.6% ( 88% - 172%) ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jpountz opened a new pull request #1543: LUCENE-9378: Disable compression on binary values whose length is less than 32.
jpountz opened a new pull request #1543: URL: https://github.com/apache/lucene-solr/pull/1543 See https://issues.apache.org/jira/browse/LUCENE-9378 This commit disables compression on short binary values, and also switches from "fast" compression to "high" compression for long values. The reasoning is that "high" compression tends to insert fewer, longer back references, which makes decompression slightly faster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12823) remove clusterstate.json in Lucene/Solr 8.0
[ https://issues.apache.org/jira/browse/SOLR-12823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119446#comment-17119446 ] Ilan Ginzburg commented on SOLR-12823: -- I rebased the PR due to a minor doc conflict. Is there anything specific that needs to happen before this gets merged or feedback given that the PR doesn't cut it? Anything I can do to speed things up? > remove clusterstate.json in Lucene/Solr 8.0 > --- > > Key: SOLR-12823 > URL: https://issues.apache.org/jira/browse/SOLR-12823 > Project: Solr > Issue Type: Task >Reporter: Varun Thacker >Priority: Major > Time Spent: 3h 40m > Remaining Estimate: 0h > > clusterstate.json is an artifact of a pre 5.0 Solr release. We should remove > that in 8.0 > It stays empty unless you explicitly ask to create the collection with the > old "stateFormat" and there is no reason for one to create a collection with > the old stateFormat. > We should also remove the "stateFormat" argument in create collection > We should also remove MIGRATESTATEVERSION as well > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14280) SolrConfig logging not helpful
[
https://issues.apache.org/jira/browse/SOLR-14280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119433#comment-17119433
]
Andras Salamon commented on SOLR-14280:
---
[~erickerickson] The two new jiras: LUCENE-9389 and SOLR-14523. I'm using
IntelliJ. Can you help me with the gradle and idea integration part?
> SolrConfig logging not helpful
> --
>
> Key: SOLR-14280
> URL: https://issues.apache.org/jira/browse/SOLR-14280
> Project: Solr
> Issue Type: Bug
>Reporter: Andras Salamon
>Assignee: Jason Gerlowski
>Priority: Minor
> Fix For: master (9.0), 8.6
>
> Attachments: SOLR-14280-01.patch, SOLR-14280-02.patch, getmessages.txt
>
>
> SolrConfig prints out a warning message if it's not able to add files to the
> classpath, but this message is not too helpful:
> {noformat}
> o.a.s.c.SolrConfig Couldn't add files from
> /opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.1850855/lib/solr/dist filtered
> by solr-langid-\d.*\.jar to classpath:
> /opt/cloudera/parcels/CDH-7.1.1-1.cdh7.1.1.p0.1850855/lib/solr/
> dist {noformat}
> The reason should be at the end of the log message but it's just repeats the
> problematic file name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14523) Enhance gradle logging calls validation: eliminate getMessage()
Andras Salamon created SOLR-14523:
-
Summary: Enhance gradle logging calls validation: eliminate
getMessage()
Key: SOLR-14523
URL: https://issues.apache.org/jira/browse/SOLR-14523
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Andras Salamon
SOLR-14280 fixed a logging problem in SolrConfig by removing a few getMessage()
calls. We could enhance this solution by modifying gradle's logging calls
validation and forbid getMessage() calls during logging. We should check the
existing code and eliminate such calls.
It is possible to suppress the warning using {{//logok}}.
[~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9389) Enhance grade logging calls validation: eliminate getMessage()
Andras Salamon created LUCENE-9389:
--
Summary: Enhance grade logging calls validation: eliminate
getMessage()
Key: LUCENE-9389
URL: https://issues.apache.org/jira/browse/LUCENE-9389
Project: Lucene - Core
Issue Type: Improvement
Reporter: Andras Salamon
SOLR-14280 fixed a logging problem in SolrConfig by removing a few getMessage()
calls. We could enhance this solution by modifying gradle's logging calls
validation and forbid getMessage() calls during logging. We should check the
existing code and eliminate such calls.
It is possible to suppress the warning using {{//logok}}.
[~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9389) Enhance gradle logging calls validation: eliminate getMessage()
[
https://issues.apache.org/jira/browse/LUCENE-9389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andras Salamon updated LUCENE-9389:
---
Summary: Enhance gradle logging calls validation: eliminate getMessage()
(was: Enhance grade logging calls validation: eliminate getMessage())
> Enhance gradle logging calls validation: eliminate getMessage()
> ---
>
> Key: LUCENE-9389
> URL: https://issues.apache.org/jira/browse/LUCENE-9389
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Andras Salamon
>Priority: Minor
>
> SOLR-14280 fixed a logging problem in SolrConfig by removing a few
> getMessage() calls. We could enhance this solution by modifying gradle's
> logging calls validation and forbid getMessage() calls during logging. We
> should check the existing code and eliminate such calls.
> It is possible to suppress the warning using {{//logok}}.
> [~erickerickson] [~gerlowskija]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9388) Move live docs to disk?
Adrien Grand created LUCENE-9388: Summary: Move live docs to disk? Key: LUCENE-9388 URL: https://issues.apache.org/jira/browse/LUCENE-9388 Project: Lucene - Core Issue Type: Improvement Reporter: Adrien Grand Live docs are the last file format whose memory usage is a function of maxDoc, let's look into moving them to disk? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14522) Error responses are always HTML
Jan Høydahl created SOLR-14522:
--
Summary: Error responses are always HTML
Key: SOLR-14522
URL: https://issues.apache.org/jira/browse/SOLR-14522
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Jan Høydahl
Solr always returns an error response as HTML
{code:java}
$ curl -i https://solr:8983/solr/admin/info/health
HTTP/2 401
content-type: text/html;charset=iso-8859-1
Error 401 Require authentication
HTTP ERROR 401 Require authentication
URI:/solr/admin/info/health
STATUS:401
MESSAGE:Require authentication
SERVLET:default
{code}
This is because Solr relies on Jetty's default ErrorHandler.
It's not a big problem, since parsing the http code is normally enough, but
this HTML tends to sneak into log messages and exceptions, and when you some
times need to parse details to act on the root cause, parsing HTML is not a
friendly format.
Proposal is to override error handler and return a well defined JSON instead,
perhpaps similar to Solr's existing error response format:
{code:java}
{
"responseHeader":{
"status":401,
"QTime":4},
"error":{
"metadata":[
"error-class","my.class",
"root-error-class","root.class"],
"msg":"my message",
"code":404}} {code}
We could optionally look at {{Accept:}} header and/or {{wt=}} request param to
determine whether to output something else than JSON.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12543) Export Handler errors come back with HTTP 200
[
https://issues.apache.org/jira/browse/SOLR-12543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119370#comment-17119370
]
Jan Høydahl commented on SOLR-12543:
Is this still the case?
> Export Handler errors come back with HTTP 200
> -
>
> Key: SOLR-12543
> URL: https://issues.apache.org/jira/browse/SOLR-12543
> Project: Solr
> Issue Type: Bug
> Components: Export Writer
>Reporter: Varun Thacker
>Priority: Major
> Labels: export-writer
>
> If I do a wrong export query like this we get back http code 200
> {code:java}
> ~/solr-7.4.0$ curl -v
> "http://localhost:8983/solr/gettingstarted/export/?q=*:*;
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 8983 (#0)
> > GET /solr/gettingstarted/export/?q=*:* HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.54.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: json
> < Transfer-Encoding: chunked
> <
> {
> "responseHeader":{"status":400},
> "response":{
> "numFound":0,
> * Connection #0 to host localhost left intact
> "docs":[{"EXCEPTION":"org.apache.solr.search.SyntaxError: No sort criteria
> was provided."}]}}
> {code}
>
> For reference if I do a bad solr query like this we get back a HTTP 400
> {code:java}
> ~/solr-7.4.0$ curl -v http://localhost:8983/solr/gettingstarted/query/?q=a:a
> * Trying ::1...
> * TCP_NODELAY set
> * Connected to localhost (::1) port 8983 (#0)
> > GET /solr/gettingstarted/query/?q=a:a HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.54.0
> > Accept: */*
> >
> < HTTP/1.1 400 Bad Request
> < Cache-Control: no-cache, no-store
> < Pragma: no-cache
> < Expires: Sat, 01 Jan 2000 01:00:00 GMT
> < Last-Modified: Mon, 09 Jul 2018 13:12:51 GMT
> < ETag: "1647f2c5e7e"
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 317
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":400,
> "QTime":1,
> "params":{
> "q":"a:a"}},
> "error":{
> "metadata":[
> "error-class","org.apache.solr.common.SolrException",
> "root-error-class","org.apache.solr.common.SolrException"],
> "msg":"undefined field a",
> "code":400}}
> * Connection #0 to host localhost left intact{code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-6595) Improve error response in case distributed collection cmd fails
[
https://issues.apache.org/jira/browse/SOLR-6595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119367#comment-17119367
]
Jan Høydahl commented on SOLR-6595:
---
Started investigating how we could return JSON error responses from Solr
instead of the default Jetty HTML error response. Then I bumped over this issue
again.
To me the proposed action sounds a whole lot better than sneaky OK responses..
> Improve error response in case distributed collection cmd fails
> ---
>
> Key: SOLR-6595
> URL: https://issues.apache.org/jira/browse/SOLR-6595
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 4.10
> Environment: SolrCloud with Client SSL
>Reporter: Sindre Fiskaa
>Priority: Minor
> Attachments: SOLR-6595.patch
>
>
> Followed the description
> https://cwiki.apache.org/confluence/display/solr/Enabling+SSL and generated a
> self signed key pair. Configured a few solr-nodes and used the collection api
> to crate a new collection. -I get error message when specify the nodes with
> the createNodeSet param. When I don't use createNodeSet param the collection
> gets created without error on random nodes. Could this be a bug related to
> the createNodeSet param?- *Update: It failed due to what turned out to be
> invalid client certificate on the overseer, and returned the following
> response:*
> {code:xml}
>
> 0 name="QTime">185
>
> org.apache.solr.client.solrj.SolrServerException:IOException occured
> when talking to server at: https://vt-searchln04:443/solr
>
>
> {code}
> *Update: Three problems:*
> # Status=0 when the cmd did not succeed (only ZK was updated, but cores not
> created due to failing to connect to shard nodes to talk to core admin API).
> # The error printed does not tell which action failed. Would be helpful to
> either get the msg from the original exception or at least some message
> saying "Failed to create core, see log on Overseer
> # State of collection is not clean since it exists as far as ZK is concerned
> but cores not created. Thus retrying the CREATECOLLECTION cmd would fail.
> Should Overseer detect error in distributed cmds and rollback changes already
> made in ZK?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14347) Autoscaling placement wrong when concurrent replica placements are calculated
[
https://issues.apache.org/jira/browse/SOLR-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119116#comment-17119116
]
Ilan Ginzburg edited comment on SOLR-14347 at 5/29/20, 6:28 AM:
[~ab] I’ve created PR
[https://github.com/apache/lucene-solr/pull/1542|https://github.com/apache/lucene-solr/pull/1542]
that I believe solves the issue identified above.
Fix has two parts:
# The obvious one: in {{PolicyHelper.getReplicaLocations()}}, the new (post
placement computation) Session is returned with the {{SessionWrapper}} so that
the next use sees the assignments of the computation rather than the initial
state.
# A less obvious one: in the same method, the way the current (orig) session
is copied to create the new Session is modified to not validate collections in
Zookeeper. This validation removed from the new session everything that didn’t
make it to Zookeeper, therefore not showing assignments in progress.
{{Policy.Session.cloneToNewSession()}} contains the code doing the copy (tried
to keep it as close to the original behavior as possible).
A multithreaded collection creation test (JMeter with 40 threads looping
through creating single shard single replica collections) led to a balanced 3
nodes cluster. Before the fix there were severe imbalances (up to a single node
taking all replicas of the run).
After this fix gets merged (as well as
[https://github.com/apache/lucene-solr/pull/1504|https://github.com/apache/lucene-solr/pull/1504]
in SOLR-14462 dealing with Session creation and caching), I believe Session
management could benefit from some refactoring and simplification.
was (Author: murblanc):
[~ab] I’ve created PR
[https://github.com/apache/lucene-solr/pull/1542|http://example.com/] that I
believe solves the issue identified above.
Fix has two parts:
# The obvious one: in {{PolicyHelper.getReplicaLocations()}}, the new (post
placement computation) Session is returned with the {{SessionWrapper}} so that
the next use sees the assignments of the computation rather than the initial
state.
# A less obvious one: in the same method, the way the current (orig) session
is copied to create the new Session is modified to not validate collections in
Zookeeper. This validation removed from the new session everything that didn’t
make it to Zookeeper, therefore not showing assignments in progress.
{{Policy.Session.cloneToNewSession()}} contains the code doing the copy (tried
to keep it as close to the original behavior as possible).
A multithreaded collection creation test (JMeter with 40 threads looping
through creating single shard single replica collections) led to a balanced 3
nodes cluster. Before the fix there were severe imbalances (up to a single node
taking all replicas of the run).
After this fix gets merged (as well as
[https://github.com/apache/lucene-solr/pull/1504|http://example.com/] in
SOLR-14462 dealing with Session creation and caching), I believe Session
management could benefit from some refactoring and simplification.
> Autoscaling placement wrong when concurrent replica placements are calculated
> -
>
> Key: SOLR-14347
> URL: https://issues.apache.org/jira/browse/SOLR-14347
> Project: Solr
> Issue Type: Bug
> Components: AutoScaling
>Affects Versions: 8.5
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Critical
> Fix For: 8.6
>
> Attachments: SOLR-14347.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Steps to reproduce:
> * create a cluster of a few nodes (tested with 7 nodes)
> * define per-collection policies that distribute replicas exclusively on
> different nodes per policy
> * concurrently create a few collections, each using a different policy
> * resulting replica placement will be seriously wrong, causing many policy
> violations
> Running the same scenario but instead creating collections sequentially
> results in no violations.
> I suspect this is caused by incorrect locking level for all collection
> operations (as defined in {{CollectionParams.CollectionAction}}) that create
> new replica placements - i.e. CREATE, ADDREPLICA, MOVEREPLICA, DELETENODE,
> REPLACENODE, SPLITSHARD, RESTORE, REINDEXCOLLECTION. All of these operations
> use the policy engine to create new replica placements, and as a result they
> change the cluster state. However, currently these operations are locked (in
> {{OverseerCollectionMessageHandler.lockTask}} ) using
> {{LockLevel.COLLECTION}}. In practice this means that the lock is held only
> for the particular collection that is being modified.
> A straightforward fix for this issue is to change the locking level to
> CLUSTER (and I confirm this fixes the scenario
[jira] [Comment Edited] (SOLR-14347) Autoscaling placement wrong when concurrent replica placements are calculated
[
https://issues.apache.org/jira/browse/SOLR-14347?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17119116#comment-17119116
]
Ilan Ginzburg edited comment on SOLR-14347 at 5/29/20, 6:27 AM:
[~ab] I’ve created PR
[https://github.com/apache/lucene-solr/pull/1542|http://example.com/] that I
believe solves the issue identified above.
Fix has two parts:
# The obvious one: in {{PolicyHelper.getReplicaLocations()}}, the new (post
placement computation) Session is returned with the {{SessionWrapper}} so that
the next use sees the assignments of the computation rather than the initial
state.
# A less obvious one: in the same method, the way the current (orig) session
is copied to create the new Session is modified to not validate collections in
Zookeeper. This validation removed from the new session everything that didn’t
make it to Zookeeper, therefore not showing assignments in progress.
{{Policy.Session.cloneToNewSession()}} contains the code doing the copy (tried
to keep it as close to the original behavior as possible).
A multithreaded collection creation test (JMeter with 40 threads looping
through creating single shard single replica collections) led to a balanced 3
nodes cluster. Before the fix there were severe imbalances (up to a single node
taking all replicas of the run).
After this fix gets merged (as well as
[https://github.com/apache/lucene-solr/pull/1504|http://example.com/] in
SOLR-14462 dealing with Session creation and caching), I believe Session
management could benefit from some refactoring and simplification.
was (Author: murblanc):
[~ab] I’ve created PR
[https://github.com/apache/lucene-solr/pull/1542|http://example.com] that I
believe solves the issue identified above.
Fix has two parts:
# The obvious one: in {{PolicyHelper.getReplicaLocations()}}, the new (post
placement computation) Session is returned with the {{SessionWrapper}} so that
the next use sees the assignments of the computation rather than the initial
state.
# A less obvious one: in the same method, the way the current (orig) session
is copied to create the new Session is modified to not validate collections in
Zookeeper. This validation removed from the new session everything that didn’t
make it to Zookeeper, therefore not showing assignments in progress.
{{Policy.Session.cloneToNewSession()}} contains the code doing the copy (tried
to keep it as close to the original behavior as possible).
A multithreaded collection creation test (JMeter with 40 threads looping
through creating single shard single replica collections) led to a balanced 3
nodes cluster. Before the fix there were severe imbalances (up to a single node
taking all replicas of the run).
After this fix gets merged (as well as
[https://github.com/apache/lucene-solr/pull/1504|http://example.com] in
SOLR-14462 dealing with Session creation and caching), I believe Session
management could benefit from some refactoring and simplification.
> Autoscaling placement wrong when concurrent replica placements are calculated
> -
>
> Key: SOLR-14347
> URL: https://issues.apache.org/jira/browse/SOLR-14347
> Project: Solr
> Issue Type: Bug
> Components: AutoScaling
>Affects Versions: 8.5
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Critical
> Fix For: 8.6
>
> Attachments: SOLR-14347.patch
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Steps to reproduce:
> * create a cluster of a few nodes (tested with 7 nodes)
> * define per-collection policies that distribute replicas exclusively on
> different nodes per policy
> * concurrently create a few collections, each using a different policy
> * resulting replica placement will be seriously wrong, causing many policy
> violations
> Running the same scenario but instead creating collections sequentially
> results in no violations.
> I suspect this is caused by incorrect locking level for all collection
> operations (as defined in {{CollectionParams.CollectionAction}}) that create
> new replica placements - i.e. CREATE, ADDREPLICA, MOVEREPLICA, DELETENODE,
> REPLACENODE, SPLITSHARD, RESTORE, REINDEXCOLLECTION. All of these operations
> use the policy engine to create new replica placements, and as a result they
> change the cluster state. However, currently these operations are locked (in
> {{OverseerCollectionMessageHandler.lockTask}} ) using
> {{LockLevel.COLLECTION}}. In practice this means that the lock is held only
> for the particular collection that is being modified.
> A straightforward fix for this issue is to change the locking level to
> CLUSTER (and I confirm this fixes the scenario described above). However,
> this effectively serializes
