[jira] [Commented] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973090#comment-16973090
]
Lucene/Solr QA commented on LUCENE-9042:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s{color}
| {color:red} lucene_grouping generated 32 new + 108 unchanged - 0 fixed = 140
total (was 108) {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 0m 17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 0m 17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 0m 17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
20s{color} | {color:green} grouping in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 2m 24s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | LUCENE-9042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985639/LUCENE-9042.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-LUCENE-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 550c7296b6f |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| javac |
https://builds.apache.org/job/PreCommit-LUCENE-Build/226/artifact/out/diff-compile-javac-lucene_grouping.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-LUCENE-Build/226/testReport/ |
| modules | C: lucene/grouping U: lucene/grouping |
| Console output |
https://builds.apache.org/job/PreCommit-LUCENE-Build/226/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9027) SIMD-based decoding of postings lists
[
https://issues.apache.org/jira/browse/LUCENE-9027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973070#comment-16973070
]
Paul Masurel commented on LUCENE-9027:
--
The benefit is probably greater on analytics.
Is there a benchmark for the ELK stack somewhere?
> SIMD-based decoding of postings lists
> -
>
> Key: LUCENE-9027
> URL: https://issues.apache.org/jira/browse/LUCENE-9027
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Adrien Grand
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [~rcmuir] has been mentioning the idea for quite some time that we might be
> able to write the decoding logic in such a way that Java would use SIMD
> instructions. More recently [~paul.masurel] wrote a [blog
> post|https://fulmicoton.com/posts/bitpacking/] that raises the point that
> Lucene could still do decode multiple ints at once in a single instruction by
> packing two ints in a long and we've had some discussions about what we could
> try in Lucene to speed up the decoding of postings. This made me want to look
> a bit deeper at what we could do.
> Our current decoding logic reads data in a byte[] and decodes packed integers
> from it. Unfortunately it doesn't make use of SIMD instructions and looks
> like
> [this|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/NaiveByteDecoder.java].
> I confirmed by looking at the generated assembly that if I take an array of
> integers and shift them all by the same number of bits then Java will use
> SIMD instructions to shift multiple integers at once. This led me to writing
> this
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SimpleSIMDDecoder.java]
> that tries as much as possible to shift long sequences of ints by the same
> number of bits to speed up decoding. It is indeed faster than the current
> logic we have, up to about 2x faster for some numbers of bits per value.
> Currently the best
> [implementation|https://github.com/jpountz/decode-128-ints-benchmark/blob/master/src/main/java/jpountz/SIMDDecoder.java]
> I've been able to come up with combines the above idea with the idea that
> Paul mentioned in his blog that consists of emulating SIMD from Java by
> packing multiple integers into a long: 2 ints, 4 shorts or 8 bytes. It is a
> bit harder to read but gives another speedup on top of the above
> implementation.
> I have a [JMH
> benchmark|https://github.com/jpountz/decode-128-ints-benchmark/] available in
> case someone would like to play with this and maybe even come up with an even
> faster implementation. It is 2-2.5x faster than our current implementation
> for most numbers of bits per value. I'm copying results here:
> {noformat}
> * `readLongs` just reads 2*bitsPerValue longs from the ByteBuffer, it serves
> as
>a baseline.
> * `decodeNaiveFromBytes` reads a byte[] and decodes from it. This is what the
>current Lucene codec does.
> * `decodeNaiveFromLongs` decodes from longs on the fly.
> * `decodeSimpleSIMD` is a simple implementation that relies on how Java
>recognizes some patterns and uses SIMD instructions.
> * `decodeSIMD` is a more complex implementation that both relies on the C2
>compiler to generate SIMD instructions and encodes 8 bytes, 4 shorts or
>2 ints in a long in order to decompress multiple values at once.
> Benchmark (bitsPerValue) (byteOrder)
> Mode Cnt Score Error Units
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 LE
> thrpt5 12.912 ± 0.393 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 1 BE
> thrpt5 12.862 ± 0.395 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 LE
> thrpt5 13.040 ± 1.162 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 2 BE
> thrpt5 13.027 ± 0.270 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 LE
> thrpt5 12.409 ± 0.637 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 3 BE
> thrpt5 12.268 ± 0.947 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 LE
> thrpt5 14.177 ± 2.263 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 4 BE
> thrpt5 11.457 ± 0.150 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 5 LE
> thrpt5 10.988 ± 1.179 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 5 BE
> thrpt5 11.226 ± 0.088 ops/us
> PackedIntsDecodeBenchmark.decodeNaiveFromBytes 6 LE
> thrpt5
[jira] [Commented] (SOLR-13821) Package Store
[
https://issues.apache.org/jira/browse/SOLR-13821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973096#comment-16973096
]
ASF subversion and git services commented on SOLR-13821:
Commit 086bad30f0007ff7e7c552bc075ff34ee0466087 in lucene-solr's branch
refs/heads/branch_8x from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=086bad3 ]
SOLR-13821: Return the size of the file
> Package Store
> -
>
> Key: SOLR-13821
> URL: https://issues.apache.org/jira/browse/SOLR-13821
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.4
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Package store is a storage managed by Solr that holds the package artifacts.
> This is replicated across nodes.
> Design is here:
> [https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?ts=5d86a8ad#]
> The package store is powered by an underlying filestore. This filestore is a
> fully replicated p2p filesystem storage for artifacts.
> The APIs are as follows
> {code:java}
> # add a file
> POST /api/cluster/files/path/to/file.jar
> #retrieve a file
> GET /api/cluster/files/path/to/file.jar
> #list files in the /path/to directory
> GET /api/cluster/files/path/to
> #GET meta info of the jar
> GET /api/cluster/files/path/to/file.jar?meta=true
> {code}
> This store keeps 2 files per file
> # The actual file say {{myplugin.jar}}
> # A metadata file {{.myplugin.jar.json}} in the same directory
> The contenbts of the metadata file is
> {code:json}
> {
> "sha512" : ""
> "sig": {
> "" :""
> }}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Closed] (SOLR-13518) improve SolrInfoBeanTest.testCallMBeanInfo debuggability
[
https://issues.apache.org/jira/browse/SOLR-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christine Poerschke closed SOLR-13518.
--
> improve SolrInfoBeanTest.testCallMBeanInfo debuggability
>
>
> Key: SOLR-13518
> URL: https://issues.apache.org/jira/browse/SOLR-13518
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Fix For: master (9.0), 8.2
>
>
> Today
> https://builds.apache.org/job/PreCommit-SOLR-Build/407/testReport/org.apache.solr/SolrInfoBeanTest/testCallMBeanInfo/
> reported
> {code}
> java.lang.AssertionError
> at
> __randomizedtesting.SeedInfo.seed([D2ED2D454B87B637:2D8BA07920FFCB29]:0)
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at org.junit.Assert.assertNotNull(Assert.java:722)
> at
> org.apache.solr.SolrInfoBeanTest.testCallMBeanInfo(SolrInfoBeanTest.java:73)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> ...
> {code}
> and since the test runs logic for multiple classes it would be nice to
> include e.g. the class name info in the assert itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-6203) cast exception while searching with sort function and result grouping
[
https://issues.apache.org/jira/browse/SOLR-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972242#comment-16972242
]
Christine Poerschke commented on SOLR-6203:
---
Unassigning the ticket to try and make it clearer that it's fully up-for-grabs
(and that I'm unlikely to have bandwidth for it in the foreseeable future,
sorry).
> cast exception while searching with sort function and result grouping
> -
>
> Key: SOLR-6203
> URL: https://issues.apache.org/jira/browse/SOLR-6203
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: 4.7, 4.8
>Reporter: Nate Dire
>Assignee: Christine Poerschke
>Priority: Major
> Attachments: README, SOLR-6203-unittest.patch,
> SOLR-6203-unittest.patch, SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch,
> SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch,
> SOLR-6203.patch, SOLR-6203.patch
>
>
> After upgrading from 4.5.1 to 4.7+, a schema including a {{"*"}} dynamic
> field as text gets a cast exception when using a sort function and result
> grouping.
> Repro (with example config):
> # Add {{"*"}} dynamic field as a {{TextField}}, eg:
> {noformat}
>
> {noformat}
> # Create sharded collection
> {noformat}
> curl
> 'http://localhost:8983/solr/admin/collections?action=CREATE=test=2=2'
> {noformat}
> # Add example docs (query must have some results)
> # Submit query which sorts on a function result and uses result grouping:
> {noformat}
> {
> "responseHeader": {
> "status": 500,
> "QTime": 50,
> "params": {
> "sort": "sqrt(popularity) desc",
> "indent": "true",
> "q": "*:*",
> "_": "1403709010008",
> "group.field": "manu",
> "group": "true",
> "wt": "json"
> }
> },
> "error": {
> "msg": "java.lang.Double cannot be cast to
> org.apache.lucene.util.BytesRef",
> "code": 500
> }
> }
> {noformat}
> Source exception from log:
> {noformat}
> ERROR - 2014-06-25 08:10:10.055; org.apache.solr.common.SolrException;
> java.lang.ClassCastException: java.lang.Double cannot be cast to
> org.apache.lucene.util.BytesRef
> at
> org.apache.solr.schema.FieldType.marshalStringSortValue(FieldType.java:981)
> at org.apache.solr.schema.TextField.marshalSortValue(TextField.java:176)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.serializeSearchGroup(SearchGroupsResultTransformer.java:125)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:65)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:43)
> at
> org.apache.solr.search.grouping.CommandHandler.processResult(CommandHandler.java:193)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:340)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> ...
> {noformat}
> It looks like {{serializeSearchGroup}} is matching the sort expression as the
> {{"*"}} dynamic field, which is a TextField in the repro.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13518) improve SolrInfoBeanTest.testCallMBeanInfo debuggability
[
https://issues.apache.org/jira/browse/SOLR-13518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christine Poerschke resolved SOLR-13518.
Resolution: Fixed
> improve SolrInfoBeanTest.testCallMBeanInfo debuggability
>
>
> Key: SOLR-13518
> URL: https://issues.apache.org/jira/browse/SOLR-13518
> Project: Solr
> Issue Type: Task
>Reporter: Christine Poerschke
>Assignee: Christine Poerschke
>Priority: Minor
> Fix For: master (9.0), 8.2
>
>
> Today
> https://builds.apache.org/job/PreCommit-SOLR-Build/407/testReport/org.apache.solr/SolrInfoBeanTest/testCallMBeanInfo/
> reported
> {code}
> java.lang.AssertionError
> at
> __randomizedtesting.SeedInfo.seed([D2ED2D454B87B637:2D8BA07920FFCB29]:0)
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:712)
> at org.junit.Assert.assertNotNull(Assert.java:722)
> at
> org.apache.solr.SolrInfoBeanTest.testCallMBeanInfo(SolrInfoBeanTest.java:73)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:566)
> ...
> {code}
> and since the test runs logic for multiple classes it would be nice to
> include e.g. the class name info in the assert itself.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Assigned] (SOLR-6203) cast exception while searching with sort function and result grouping
[
https://issues.apache.org/jira/browse/SOLR-6203?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christine Poerschke reassigned SOLR-6203:
-
Assignee: (was: Christine Poerschke)
> cast exception while searching with sort function and result grouping
> -
>
> Key: SOLR-6203
> URL: https://issues.apache.org/jira/browse/SOLR-6203
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: 4.7, 4.8
>Reporter: Nate Dire
>Priority: Major
> Attachments: README, SOLR-6203-unittest.patch,
> SOLR-6203-unittest.patch, SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch,
> SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch, SOLR-6203.patch,
> SOLR-6203.patch, SOLR-6203.patch
>
>
> After upgrading from 4.5.1 to 4.7+, a schema including a {{"*"}} dynamic
> field as text gets a cast exception when using a sort function and result
> grouping.
> Repro (with example config):
> # Add {{"*"}} dynamic field as a {{TextField}}, eg:
> {noformat}
>
> {noformat}
> # Create sharded collection
> {noformat}
> curl
> 'http://localhost:8983/solr/admin/collections?action=CREATE=test=2=2'
> {noformat}
> # Add example docs (query must have some results)
> # Submit query which sorts on a function result and uses result grouping:
> {noformat}
> {
> "responseHeader": {
> "status": 500,
> "QTime": 50,
> "params": {
> "sort": "sqrt(popularity) desc",
> "indent": "true",
> "q": "*:*",
> "_": "1403709010008",
> "group.field": "manu",
> "group": "true",
> "wt": "json"
> }
> },
> "error": {
> "msg": "java.lang.Double cannot be cast to
> org.apache.lucene.util.BytesRef",
> "code": 500
> }
> }
> {noformat}
> Source exception from log:
> {noformat}
> ERROR - 2014-06-25 08:10:10.055; org.apache.solr.common.SolrException;
> java.lang.ClassCastException: java.lang.Double cannot be cast to
> org.apache.lucene.util.BytesRef
> at
> org.apache.solr.schema.FieldType.marshalStringSortValue(FieldType.java:981)
> at org.apache.solr.schema.TextField.marshalSortValue(TextField.java:176)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.serializeSearchGroup(SearchGroupsResultTransformer.java:125)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:65)
> at
> org.apache.solr.search.grouping.distributed.shardresultserializer.SearchGroupsResultTransformer.transform(SearchGroupsResultTransformer.java:43)
> at
> org.apache.solr.search.grouping.CommandHandler.processResult(CommandHandler.java:193)
> at
> org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:340)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:218)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> ...
> {noformat}
> It looks like {{serializeSearchGroup}} is matching the sort expression as the
> {{"*"}} dynamic field, which is a TextField in the repro.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13759) Optimize Queries when query filtering by TRA router.field
[
https://issues.apache.org/jira/browse/SOLR-13759?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972329#comment-16972329
]
mosh commented on SOLR-13759:
-
Example:
Given a times based data stored in TRA (say IOT signals) - today, querying a
specific *date range* will query *all* TRA collections (rather than relevant
collections that potentially holding the desired data) then on each collection
we filter by the specified field.
If the specified fq date range date field is the router.field I propose an
optimization to today's behavior by *filtering* out irrelevant collections
before even querying them.
In HttpSolrCall#init:279:
{code:java}
collectionsList = resolveCollectionListOrAlias(queryParams.get(COLLECTION_PROP,
def));{code}
collectionsList filtering will look somewhat like the below:
{code:java}
collectionsList =
collectionsList.stream().filter(collectionName->isDateInRange(fqDateRange,
collectionName)).collect(Collectors.toList());
{code}
Using this practice we are avoiding redundant queries to collections that we
are 100% sure that doesn't store the relevant data.
> Optimize Queries when query filtering by TRA router.field
> -
>
> Key: SOLR-13759
> URL: https://issues.apache.org/jira/browse/SOLR-13759
> Project: Solr
> Issue Type: Sub-task
>Reporter: mosh
>Priority: Minor
>
> We are currently testing TRA using Solr 7.7, having >300 shards in the alias,
> with much growth in the coming months.
> The "hot" data(in our case, more recent) will be stored on stronger
> nodes(SSD, more RAM, etc).
> A proposal of optimizing queries will be by filtering query by date range, by
> that we will be able to querying the specific TRA collections taking
> advantage of the TRA mechanism of partitioning data based on date.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13922) Fix html file generation location for the Gradle build
Erick Erickson created SOLR-13922: - Summary: Fix html file generation location for the Gradle build Key: SOLR-13922 URL: https://issues.apache.org/jira/browse/SOLR-13922 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Erick Erickson Assignee: Dawid Weiss Raising and assigning as per Dawid's comment on the dev list In the gradle_7 branch the HTML files were generated in ./solr/solr-ref-guide/build, but now they are in the top-level ./build directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13921) Processing UpdateRequest with delegation token throws NullPointerException
[
https://issues.apache.org/jira/browse/SOLR-13921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972315#comment-16972315
]
Lucene/Solr QA commented on SOLR-13921:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 45m
8s{color} | {color:green} core in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 5m
33s{color} | {color:green} solrj in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 55m 5s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13921 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985560/SOLR-13921.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 88477112128 |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/594/testReport/ |
| modules | C: solr/core solr/solrj U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/594/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Processing UpdateRequest with delegation token throws NullPointerException
> --
>
> Key: SOLR-13921
> URL: https://issues.apache.org/jira/browse/SOLR-13921
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrJ
>Affects Versions: 7.4, 8.3
>Reporter: Istvan Farkas
>Priority: Minor
> Attachments: SOLR-13921.patch
>
>
> When sending UpdateRequests with delegation tokens to Solr using SolrJ, the
> createMethod of DelegationTokenHttpSolrClient will throw a
> NullPointerException:
>
> {code:java}
> [junit4] ERROR 3.41s |
> TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests
> <<<
>[junit4]> Throwable #1: java.lang.NullPointerException
>[junit4]> at
> __randomizedtesting.SeedInfo.seed([B9AE8E4E0CDF1B3D:DBA0B722C813061D]:0)
>[junit4]> at
> org.apache.solr.client.solrj.impl.DelegationTokenHttpSolrClient.createMethod(DelegationTokenHttpSolrClient.java:93)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:258)
>[junit4]> at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:249)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.doSolrRequest(TestSolrCloudWithDelegationTokens.java:246)
>[junit4]> at
> org.apache.solr.cloud.TestSolrCloudWithDelegationTokens.testDelegationTokenSolrClientWithUpdateRequests(TestSolrCloudWithDelegationTokens.java:477)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>[junit4]> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>[junit4]> at
>
[jira] [Assigned] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski reassigned SOLR-13890:
--
Assignee: Jason Gerlowski
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8920) Reduce size of FSTs due to use of direct-addressing encoding
[ https://issues.apache.org/jira/browse/LUCENE-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972390#comment-16972390 ] Adrien Grand commented on LUCENE-8920: -- Yes, doing this in a separate JIRA sounds like a good idea to me. I reviewed the change yesterday and it looked good to me. I plan to merge tomorrow if there are no objections. > Reduce size of FSTs due to use of direct-addressing encoding > - > > Key: LUCENE-8920 > URL: https://issues.apache.org/jira/browse/LUCENE-8920 > Project: Lucene - Core > Issue Type: Improvement >Reporter: Michael Sokolov >Priority: Minor > Attachments: TestTermsDictRamBytesUsed.java > > Time Spent: 4h 20m > Remaining Estimate: 0h > > Some data can lead to worst-case ~4x RAM usage due to this optimization. > Several ideas were suggested to combat this on the mailing list: > bq. I think we can improve thesituation here by tracking, per-FST instance, > the size increase we're seeing while building (or perhaps do a preliminary > pass before building) in order to decide whether to apply the encoding. > bq. we could also make the encoding a bit more efficient. For instance I > noticed that arc metadata is pretty large in some cases (in the 10-20 bytes) > which make gaps very costly. Associating each label with a dense id and > having an intermediate lookup, ie. lookup label -> id and then id->arc offset > instead of doing label->arc directly could save a lot of space in some cases? > Also it seems that we are repeating the label in the arc metadata when > array-with-gaps is used, even though it shouldn't be necessary since the > label is implicit from the address? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9018) Separator for ConcatenateGraphFilterFactory
[ https://issues.apache.org/jira/browse/LUCENE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972335#comment-16972335 ] Stanislav Mikulchik commented on LUCENE-9018: - > There is a backwards-compatibility concern we overlooked Added getLuceneMatchVersion() check to ConcatenateGraphFilterFactory > getCharacter on an empty string can mean null Done > RE CompletionAnalyzer Reverted changes [^LUCENE-9018.patch] > Separator for ConcatenateGraphFilterFactory > --- > > Key: LUCENE-9018 > URL: https://issues.apache.org/jira/browse/LUCENE-9018 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis >Reporter: Stanislav Mikulchik >Assignee: David Smiley >Priority: Minor > Attachments: LUCENE-9018.patch, LUCENE-9018.patch, LUCENE-9018.patch > > > I would like to have an option to choose a separator to use for token > concatenation. Currently ConcatenateGraphFilterFactory can use only "\u001F" > symbol. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9018) Separator for ConcatenateGraphFilterFactory
[ https://issues.apache.org/jira/browse/LUCENE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stanislav Mikulchik updated LUCENE-9018: Attachment: LUCENE-9018.patch > Separator for ConcatenateGraphFilterFactory > --- > > Key: LUCENE-9018 > URL: https://issues.apache.org/jira/browse/LUCENE-9018 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis >Reporter: Stanislav Mikulchik >Assignee: David Smiley >Priority: Minor > Attachments: LUCENE-9018.patch, LUCENE-9018.patch, LUCENE-9018.patch > > > I would like to have an option to choose a separator to use for token > concatenation. Currently ConcatenateGraphFilterFactory can use only "\u001F" > symbol. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972740#comment-16972740
]
Andrzej Bialecki commented on SOLR-13898:
-
Yeah, at this point I'm not very concerned about legacy caches, but until they
are removed I'd prefer them to at least behave correctly, even if they're not
the most efficient.
I briefly looked at how caches are accessed and populated in
{{SolrIndexSearcher}} and I'm not comfortable touching this code, except in a
few obvious places (I'll update the patch in a moment).
I noticed that {{UnifiedHighlighter}} uses {{SolrIndexSearcher.doc(id,
visitor)}}, which in turn delegates to {{SolrDocumentFetcher.doc(id,
visitor)}}, which does not use cache. Is this on purpose? I would expect
highlighting to benefit from caching ...
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345019129 ## File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackage.java ## @@ -0,0 +1,107 @@ +package org.apache.solr.packagemanager; + + +import java.io.Serializable; +import java.util.Date; +import java.util.List; +import java.util.Map; + +import com.fasterxml.jackson.annotation.JsonProperty; Review comment: Please take a look at `TestSolrJacksonAnnotation` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345017318
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageRepository.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Map;
+
+import org.apache.solr.common.SolrException;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Abstract class for a repository, holding {@link SolrPackage} items.
+ */
+public abstract class PackageRepository {
+
+ @JsonProperty("name")
Review comment:
can we use `org.apache.solr.common.annotation.Jsonproperty` here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345017725
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipFile;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.lucene.util.SuppressForbidden;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.zafarkhaja.semver.Version;
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.Configuration;
+import com.jayway.jsonpath.spi.json.JacksonJsonProvider;
+import com.jayway.jsonpath.spi.json.JsonProvider;
+import com.jayway.jsonpath.spi.mapper.JacksonMappingProvider;
+import com.jayway.jsonpath.spi.mapper.MappingProvider;
+
+public class PackageUtils {
+
+ public static Configuration jsonPathConfiguration() {
+MappingProvider provider = new JacksonMappingProvider();
+JsonProvider jsonProvider = new JacksonJsonProvider();
+Configuration c =
Configuration.builder().jsonProvider(jsonProvider).mappingProvider(provider).options(com.jayway.jsonpath.Option.REQUIRE_PROPERTIES).build();
+return c;
+ }
+
+ /**
+ * Uploads a file to the package store / file store of Solr.
+ *
+ * @param client A Solr client
+ * @param buffer File contents
+ * @param name Name of the file as it will appear in the file store (can be
hierarchical)
+ * @param sig Signature digest (public key should be separately uploaded to
ZK)
+ */
+ public static void postFile(SolrClient client, ByteBuffer buffer, String
name, String sig)
+ throws SolrServerException, IOException {
+String resource = "/api/cluster/files" + name;
+ModifiableSolrParams params = new ModifiableSolrParams();
+if (sig != null) {
+ params.add("sig", sig);
+}
+V2Response rsp = new V2Request.Builder(resource)
+.withMethod(SolrRequest.METHOD.PUT)
+.withPayload(buffer)
+.forceV2(true)
Review comment:
forceV2() is only required in tests
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] noblepaul commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
noblepaul commented on a change in pull request #994: SOLR-13662: Package
Manager (CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345019280
##
File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackage.java
##
@@ -0,0 +1,107 @@
+package org.apache.solr.packagemanager;
+
+
+import java.io.Serializable;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Describes a package (along with all released versions) as it appears in a
repository.
+ */
+public class SolrPackage implements Serializable, Comparable {
+
+ public String id;
+ public String description;
+ public List versions;
+
+ private String repositoryId;
+
+ public static class SolrPackageRelease {
+
+public String version;
+public Date date;
+public String url;
+
+public String sha512sum;
+public String sig;
+
+public Metadata metadata;
+@Override
+public String toString() {
Review comment:
`ReflectMapWriter` only works if the fields are annotated using
`org.apache.solr.common.annotation.JsonProperty`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972743#comment-16972743
]
Andrzej Bialecki commented on SOLR-13898:
-
Updated patch:
* modify {{SolrIndexSearcher}} {{getDocSet(Query)}} , {{getDocSet(Query,
Filter)}} and {{getPositiveDocSet(Query)}} to use {{computeIfAbsent}}.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972743#comment-16972743
]
Andrzej Bialecki edited comment on SOLR-13898 at 11/12/19 8:18 PM:
---
Updated patch:
* modify {{SolrIndexSearcher}} {{getDocSet(Query)}} , {{getDocSet(Query,
Filter)}} and {{getPositiveDocSet(Query)}} to use {{computeIfAbsent}}.
All tests are passing. If there are no objections I'd like to finish this issue
at this stage and track further improvements in new Jira issues.
was (Author: ab):
Updated patch:
* modify {{SolrIndexSearcher}} {{getDocSet(Query)}} , {{getDocSet(Query,
Filter)}} and {{getPositiveDocSet(Query)}} to use {{computeIfAbsent}}.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972758#comment-16972758
]
Chris M. Hostetter edited comment on SOLR-13864 at 11/12/19 9:06 PM:
-
-Commit 603be023feaf3f8e3e739e532b488068710d9097 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter-
-[ [https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02] ]-
-SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated-
Mistaken Jira tagging in commit message
was (Author: jira-bot):
Commit 603be023feaf3f8e3e739e532b488068710d9097 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02 ]
SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13925) Maven build doesn't include clusterstate.json in jar
[ https://issues.apache.org/jira/browse/SOLR-13925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972806#comment-16972806 ] Justin Deoliveira commented on SOLR-13925: -- Pull request in github: [https://github.com/apache/lucene-solr/pull/1006] > Maven build doesn't include clusterstate.json in jar > > > Key: SOLR-13925 > URL: https://issues.apache.org/jira/browse/SOLR-13925 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrJ >Affects Versions: 8.3 >Reporter: Justin Deoliveira >Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > The maven pom template for the solrj module doesn't appear to include > resources so files such as clusterstate.json don't get included in the final > jar. This results in an error during startup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] jdeolive opened a new pull request #1006: SOLR-13925: including resource directory in solrj pom template
jdeolive opened a new pull request #1006: SOLR-13925: including resource directory in solrj pom template URL: https://github.com/apache/lucene-solr/pull/1006 # Description Updating the pom template for the solrj module to include resources such as clusterstate.json. # Solution See description. # Tests No unit tests, wasn't evident how I could include a test of the maven artifacts. # Checklist - [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [x] I have developed this patch against the `master` branch. - [ ] I have run `ant precommit` and the appropriate test suite. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-13864:
--
Attachment: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
Status: Open (was: Open)
[~jbernste] - it looks like you only fixed testGammaDistribution ?
what about testZipFDistribution, testGeometricDistribution, and testFuzzyKmeans
? ... as mentioned above they also seem to fail sporadically due to explicit
assumptions about the underlying random distributions.
Attaching a recent jenkins failure from testZipFDistribution...
{noformat}
[junit4] 2> 288619 INFO
(TEST-MathExpressionTest.testZipFDistribution-seed#[4B489A4C6D218B8D]) [ ]
o.a.s.SolrTestCaseJ4 ###Ending testZipFDistribution
[junit4] 2> NOTE: reproduce with: ant test -Dtestcase=MathExpressionTest
-Dtests.method=testZipFDistribution -Dtests.seed=4B489A4C6D218B8D
-Dtests.multiplier=2 -Dtests.slow=true -Dtests.badapples=true
-Dtests.locale=ja-JP -Dtests.timezone=America/St_Lucia -Dtests.asserts=true
-Dtests.file.encoding=UTF-8
[junit4] ERROR 0.07s J1 | MathExpressionTest.testZipFDistribution <<<
[junit4]> Throwable #1: java.lang.Exception: Zipf distribution not
descending!!!
[junit4]>at
__randomizedtesting.SeedInfo.seed([4B489A4C6D218B8D:6FFDF7687A8983A5]:0)
[junit4]>at
org.apache.solr.client.solrj.io.stream.MathExpressionTest.testZipFDistribution(MathExpressionTest.java:3766)
{noformat}
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13869) deprecated and remove SolrTestCaseJ4.getNextAvailablePort
[
https://issues.apache.org/jira/browse/SOLR-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972776#comment-16972776
]
ASF subversion and git services commented on SOLR-13869:
Commit 3a5d67b87b2c5babfd42e822894d5a19281fed82 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=3a5d67b ]
SOLR-13869: remove deprecated SolrTestCaseJ4.getNextAvailablePort() from master
> deprecated and remove SolrTestCaseJ4.getNextAvailablePort
> -
>
> Key: SOLR-13869
> URL: https://issues.apache.org/jira/browse/SOLR-13869
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
>
> {{SolrTestCaseJ4.getNextAvailablePort()}} is a problematic method in that the
> entire premise assumes that since the OS allowed us to bind to a port once,
> it will allow us to re-bind to that port again at some future point in the
> text execution -- but that is not garunteed, as soon as
> getNextAvailablePort() releases the port assigned to {{new ServerSocket(0)}}
> the OS may allocate that port to some other process.
> We should deprecate & remove this method {{getNextAvailablePort()}} to
> prevent tests from adopting this bad pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13869) deprecated and remove SolrTestCaseJ4.getNextAvailablePort
[
https://issues.apache.org/jira/browse/SOLR-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972778#comment-16972778
]
ASF subversion and git services commented on SOLR-13869:
Commit 5df9a51cbfb51bcd1f2a723cef9f6d22accd72f1 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=5df9a51 ]
Fix SOLR-13869 typo in CHANGES.txt
> deprecated and remove SolrTestCaseJ4.getNextAvailablePort
> -
>
> Key: SOLR-13869
> URL: https://issues.apache.org/jira/browse/SOLR-13869
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
>
> {{SolrTestCaseJ4.getNextAvailablePort()}} is a problematic method in that the
> entire premise assumes that since the OS allowed us to bind to a port once,
> it will allow us to re-bind to that port again at some future point in the
> text execution -- but that is not garunteed, as soon as
> getNextAvailablePort() releases the port assigned to {{new ServerSocket(0)}}
> the OS may allocate that port to some other process.
> We should deprecate & remove this method {{getNextAvailablePort()}} to
> prevent tests from adopting this bad pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13890) Add postfilter support to {!terms} queries
[
https://issues.apache.org/jira/browse/SOLR-13890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski updated SOLR-13890:
---
Attachment: SOLR-13890.patch
Status: Open (was: Open)
Attaching a revised patch with some tests. I also tried to add some
clarification to the "terms" coverage in the ref-guide around how each "method"
scales, and when it might be good to use each one. (I don't really know
anything about the "automaton" method though...I wonder if David Smiley knows
anything about when its best to use that method. It looks like he added that
back in the day).
Anyway, this is getting closer to being commit-ready.
> Add postfilter support to {!terms} queries
> --
>
> Key: SOLR-13890
> URL: https://issues.apache.org/jira/browse/SOLR-13890
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: query parsers
>Affects Versions: master (9.0)
>Reporter: Jason Gerlowski
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13890.patch, SOLR-13890.patch
>
>
> There are some use-cases where it'd be nice if the "terms" qparser created a
> query that could be run as a postfilter. Particularly, when users are
> checking for hundreds or thousands of terms, a postfilter implementation can
> be more performant than the standard processing.
> WIth this issue, I'd like to propose a post-filter implementation for the
> {{docValuesTermsFilter}} "method". Postfilter creation can use a
> SortedSetDocValues object to populate a DV bitset with the "terms" being
> checked for. Each document run through the post-filter can look at their
> doc-values for the field in question and check them efficiently against the
> constructed bitset.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972758#comment-16972758
]
ASF subversion and git services commented on SOLR-13864:
Commit 603be023feaf3f8e3e739e532b488068710d9097 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02 ]
SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[ https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972766#comment-16972766 ] Adam Walz commented on LUCENE-9015: --- I'll set up my own email temporarily for testing. > Configure branches, auto build and auto stage/publish > - > > Key: LUCENE-9015 > URL: https://issues.apache.org/jira/browse/LUCENE-9015 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > Commit to master should build and publish the staging site > Find a simple way to trigger publishing of main site from staging -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972767#comment-16972767
]
ASF subversion and git services commented on SOLR-13864:
Commit b872863da9e06a9f21562830cbd26d24d30a8138 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b872863 ]
SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated
(cherry picked from commit 603be023feaf3f8e3e739e532b488068710d9097)
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-site] adamwalz opened a new pull request #6: LUCENE-9015 Add notification to .asf-yaml automated Pelican builds
adamwalz opened a new pull request #6: LUCENE-9015 Add notification to .asf-yaml automated Pelican builds URL: https://github.com/apache/lucene-site/pull/6 Temporarily using a...@adamwalz.net. Will change to bui...@lucene.apache.org after initial testing and after this build gets whitelisted to send to bui...@lucene.apache.org. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13922) Fix html file generation location for the Gradle build
[ https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972774#comment-16972774 ] Dawid Weiss commented on SOLR-13922: Changing it back is not a problem but the ant version is/ was generating everything to top-level ./build/solr-ref-guide directory so gradle now is actually consistent with ant (and it wasn't before). Should I still change it? > Fix html file generation location for the Gradle build > -- > > Key: SOLR-13922 > URL: https://issues.apache.org/jira/browse/SOLR-13922 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Dawid Weiss >Priority: Major > > Raising and assigning as per Dawid's comment on the dev list > In the gradle_7 branch the HTML files were generated in > ./solr/solr-ref-guide/build, but now they are in the top-level ./build > directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13869) deprecated and remove SolrTestCaseJ4.getNextAvailablePort
[
https://issues.apache.org/jira/browse/SOLR-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter resolved SOLR-13869.
---
Fix Version/s: 8.4
master (9.0)
Resolution: Fixed
> deprecated and remove SolrTestCaseJ4.getNextAvailablePort
> -
>
> Key: SOLR-13869
> URL: https://issues.apache.org/jira/browse/SOLR-13869
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Fix For: master (9.0), 8.4
>
>
> {{SolrTestCaseJ4.getNextAvailablePort()}} is a problematic method in that the
> entire premise assumes that since the OS allowed us to bind to a port once,
> it will allow us to re-bind to that port again at some future point in the
> text execution -- but that is not garunteed, as soon as
> getNextAvailablePort() releases the port assigned to {{new ServerSocket(0)}}
> the OS may allocate that port to some other process.
> We should deprecate & remove this method {{getNextAvailablePort()}} to
> prevent tests from adopting this bad pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13869) deprecated and remove SolrTestCaseJ4.getNextAvailablePort
[
https://issues.apache.org/jira/browse/SOLR-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972786#comment-16972786
]
ASF subversion and git services commented on SOLR-13869:
Commit 700aeeb58f6a61af01e324590acd183fd1a9cf22 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=700aeeb ]
Fix SOLR-13869 typo in CHANGES.txt
(cherry picked from commit 5df9a51cbfb51bcd1f2a723cef9f6d22accd72f1)
> deprecated and remove SolrTestCaseJ4.getNextAvailablePort
> -
>
> Key: SOLR-13869
> URL: https://issues.apache.org/jira/browse/SOLR-13869
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Fix For: master (9.0), 8.4
>
>
> {{SolrTestCaseJ4.getNextAvailablePort()}} is a problematic method in that the
> entire premise assumes that since the OS allowed us to bind to a port once,
> it will allow us to re-bind to that port again at some future point in the
> text execution -- but that is not garunteed, as soon as
> getNextAvailablePort() releases the port assigned to {{new ServerSocket(0)}}
> the OS may allocate that port to some other process.
> We should deprecate & remove this method {{getNextAvailablePort()}} to
> prevent tests from adopting this bad pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13925) Maven build doesn't include clusterstate.json in jar
Justin Deoliveira created SOLR-13925: Summary: Maven build doesn't include clusterstate.json in jar Key: SOLR-13925 URL: https://issues.apache.org/jira/browse/SOLR-13925 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrJ Affects Versions: 8.3 Reporter: Justin Deoliveira The maven pom template for the solrj module doesn't appear to include resources so files such as clusterstate.json don't get included in the final jar. This results in an error during startup. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13922) Fix html file generation location for the Gradle build
[
https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972804#comment-16972804
]
Dawid Weiss commented on SOLR-13922:
Either the build folder needs to be identical to ant's or we have to make
solr-root-path configurable in _config.yml.template and pass proper relative
values from gradle (and ant):
{code:java}
solr-attributes:
solr-root-path: "../../../" {code}
> Fix html file generation location for the Gradle build
> --
>
> Key: SOLR-13922
> URL: https://issues.apache.org/jira/browse/SOLR-13922
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Dawid Weiss
>Priority: Major
>
> Raising and assigning as per Dawid's comment on the dev list
> In the gradle_7 branch the HTML files were generated in
> ./solr/solr-ref-guide/build, but now they are in the top-level ./build
> directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13922) Gradle build output folder and relative adoc includes
[ https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Weiss updated SOLR-13922: --- Summary: Gradle build output folder and relative adoc includes (was: Fix html file generation location for the Gradle build) > Gradle build output folder and relative adoc includes > - > > Key: SOLR-13922 > URL: https://issues.apache.org/jira/browse/SOLR-13922 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Dawid Weiss >Priority: Major > > Raising and assigning as per Dawid's comment on the dev list > In the gradle_7 branch the HTML files were generated in > ./solr/solr-ref-guide/build, but now they are in the top-level ./build > directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972767#comment-16972767
]
Chris M. Hostetter edited comment on SOLR-13864 at 11/12/19 9:07 PM:
-
-Commit b872863da9e06a9f21562830cbd26d24d30a8138 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter-
-[ [https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b872863] ]-
-SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated-
-(cherry picked from commit 603be023feaf3f8e3e739e532b488068710d9097)-
Mistaken Jira tagging in commit message
was (Author: jira-bot):
Commit b872863da9e06a9f21562830cbd26d24d30a8138 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b872863 ]
SOLR-13864: SolrTestCaseJ4.getNextAvailablePort() has been deprecated
(cherry picked from commit 603be023feaf3f8e3e739e532b488068710d9097)
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13869) deprecated and remove SolrTestCaseJ4.getNextAvailablePort
[
https://issues.apache.org/jira/browse/SOLR-13869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972781#comment-16972781
]
Chris M. Hostetter commented on SOLR-13869:
---
I screwed up the jira tagging in the original deprecation commits...
* master: https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=603be02
* 8x: https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b872863
> deprecated and remove SolrTestCaseJ4.getNextAvailablePort
> -
>
> Key: SOLR-13869
> URL: https://issues.apache.org/jira/browse/SOLR-13869
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
>
> {{SolrTestCaseJ4.getNextAvailablePort()}} is a problematic method in that the
> entire premise assumes that since the OS allowed us to bind to a port once,
> it will allow us to re-bind to that port again at some future point in the
> text execution -- but that is not garunteed, as soon as
> getNextAvailablePort() releases the port assigned to {{new ServerSocket(0)}}
> the OS may allocate that port to some other process.
> We should deprecate & remove this method {{getNextAvailablePort()}} to
> prevent tests from adopting this bad pattern.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9015) Configure branches, auto build and auto stage/publish
[ https://issues.apache.org/jira/browse/LUCENE-9015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972792#comment-16972792 ] Jan Høydahl commented on LUCENE-9015: - Daniel said on Slack that "it should notify you on commits@lucene if I'm worth my salt". So seems like notifications are implicit. They are having some difficulties with the builder that they are looking into right now, so hopefully on our next commit there will be some action. > Configure branches, auto build and auto stage/publish > - > > Key: LUCENE-9015 > URL: https://issues.apache.org/jira/browse/LUCENE-9015 > Project: Lucene - Core > Issue Type: Sub-task >Reporter: Jan Høydahl >Assignee: Jan Høydahl >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Commit to master should build and publish the staging site > Find a simple way to trigger publishing of main site from staging -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13922) Fix html file generation location for the Gradle build
[ https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972795#comment-16972795 ] Dawid Weiss commented on SOLR-13922: Ok, it's not the same folder. Previously it was solr/build now it's top-level/build/. > Fix html file generation location for the Gradle build > -- > > Key: SOLR-13922 > URL: https://issues.apache.org/jira/browse/SOLR-13922 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Dawid Weiss >Priority: Major > > Raising and assigning as per Dawid's comment on the dev list > In the gradle_7 branch the HTML files were generated in > ./solr/solr-ref-guide/build, but now they are in the top-level ./build > directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] diegoceccarelli opened a new pull request #1005: LUCENE 9042: Refactor TopGroups.merge tests
diegoceccarelli opened a new pull request #1005: LUCENE 9042: Refactor TopGroups.merge tests URL: https://github.com/apache/lucene-solr/pull/1005 # Description LUCENE 9042: Refactor TopGroups.merge tests # Solution This task proposes a refactoring of the test coverage for the TopGroups.merge method implemented in LUCENE-9010. For now it will cover only 3 main cases. 1. Merging to empty TopGroups 2. Merging a TopGroups with scores and a TopGroups without scores (currently broken because of LUCENE-8996 bug) 3. Merging two TopGroups with scores. I'm planning to increase the coverage testing also invalid inputs but I would do that in a separate PR to keep the code readable. # Tests I'm contributing unit tests. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://wiki.apache.org/solr/HowToContribute) and my code conforms to the standards described there to the best of my ability. - [x] I have created a Jira issue and added the issue ID to my pull request title. - [x] I am authorized to contribute this code to the ASF and have removed any code I do not have a license to distribute. - [x] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [x] I have developed this patch against the `master` branch. - [x] I have run `ant precommit` and the appropriate test suite. - [x] I have added tests for my changes. - [ ] I have added documentation for the [Ref Guide](https://github.com/apache/lucene-solr/tree/master/solr/solr-ref-guide) (for Solr changes only). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13870) eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
[
https://issues.apache.org/jira/browse/SOLR-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris M. Hostetter updated SOLR-13870:
--
Fix Version/s: 8.4
master (9.0)
Resolution: Fixed
Status: Resolved (was: Patch Available)
> eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
> --
>
> Key: SOLR-13870
> URL: https://issues.apache.org/jira/browse/SOLR-13870
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Fix For: master (9.0), 8.4
>
> Attachments: SOLR-13870.patch
>
>
> {{SolrJmxReporterTest}} is currently the only solr test using (the
> problematic) {{SolrTestCaseJ4.getNextAvailablePort}} helper method in order
> to assign a RMI port for running a JMX server.
> This test doesn't actually care that it's using RMI for JMX, it only cares
> that:
> # an MBeanServer is initialized that it can query
> # that MBeanServer can be specified in the plugin configuration for the
> {{SolrJmxReporter}}
> We should cleanup the test initialization so that it no longer depends on RMI
> via the {{serviceUrl}} plugin configuration option, and instead just use the
> {{agentId}} to specify an MBeanServer bootstrapped by the test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13870) eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
[
https://issues.apache.org/jira/browse/SOLR-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972618#comment-16972618
]
ASF subversion and git services commented on SOLR-13870:
Commit 26b7799af6c298ff5478887bc12539cca4bff094 in lucene-solr's branch
refs/heads/branch_8x from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=26b7799 ]
SOLR-13870: SolrJmxReporterTest no longer depends on (buggy)
getNextAvailablePort()
(cherry picked from commit 367fe2f341cca18fb3d22e203b8cd8a91dd36948)
> eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
> --
>
> Key: SOLR-13870
> URL: https://issues.apache.org/jira/browse/SOLR-13870
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-13870.patch
>
>
> {{SolrJmxReporterTest}} is currently the only solr test using (the
> problematic) {{SolrTestCaseJ4.getNextAvailablePort}} helper method in order
> to assign a RMI port for running a JMX server.
> This test doesn't actually care that it's using RMI for JMX, it only cares
> that:
> # an MBeanServer is initialized that it can query
> # that MBeanServer can be specified in the plugin configuration for the
> {{SolrJmxReporter}}
> We should cleanup the test initialization so that it no longer depends on RMI
> via the {{serviceUrl}} plugin configuration option, and instead just use the
> {{agentId}} to specify an MBeanServer bootstrapped by the test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diego Ceccarelli updated LUCENE-9042:
-
Attachment: LUCENE-9042.patch
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
> Attachments: LUCENE-9042.patch
>
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrzej Bialecki updated SOLR-13898:
Attachment: SOLR-13898.patch
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13924) MoveReplica failures when using HDFS (NullPointerException)
Chris M. Hostetter created SOLR-13924:
-
Summary: MoveReplica failures when using HDFS
(NullPointerException)
Key: SOLR-13924
URL: https://issues.apache.org/jira/browse/SOLR-13924
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 8.3
Reporter: Chris M. Hostetter
Assignee: Shalin Shekhar Mangar
Based on recent jenkins test failures, it appears that attemping to use the
"MoveReplica" command on HDFS has a high chance of failure due to an underlying
NPE.
I'm not sure if this bug *only* affects HDFS, or if it's just more likly to
occur when using HDFS due to some timing quirks.
It's also possible that the bug impacts non-HDFS users just as much as HDFS
users, but only manifests in our tests due to some quick of our {{cloud-hdfs}}
test configs.
The problem appears to be new in 8.3 as a result of changes made in SOLR-13843
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9031) UnsupportedOperationException on highlighting Interval Query
[ https://issues.apache.org/jira/browse/LUCENE-9031?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972596#comment-16972596 ] Mikhail Khludnev commented on LUCENE-9031: -- Ok. (As I said, I will abandon this huge one https://github.com/apache/lucene-solr/pull/985) Just for the reference, I've fixed one postings MTQ highlight for intervals. Most changes seems reasonable, eg MultTermsIS is visited as a regular WildcardQuerry. One thing, which puzzled me a little https://github.com/apache/lucene-solr/pull/985/files#diff-f5538289e23aabdd53bc3bcbc59da342L239 I think submatch of disjunction is an one of the legs. But not, underneath of one of the legs, just because submatches of TermIS is null. Also, this fix have broken a lot of existing UH-MTQ tests - not good. Most of them fails in late assert while juggling passages https://github.com/apache/lucene-solr/blob/master/lucene/highlighter/src/java/org/apache/lucene/search/uhighlight/OffsetsEnum.java#L219. > UnsupportedOperationException on highlighting Interval Query > > > Key: LUCENE-9031 > URL: https://issues.apache.org/jira/browse/LUCENE-9031 > Project: Lucene - Core > Issue Type: Bug > Components: modules/queries >Reporter: Mikhail Khludnev >Assignee: Mikhail Khludnev >Priority: Major > Fix For: 8.4 > > Attachments: LUCENE-9031.patch, LUCENE-9031.patch, LUCENE-9031.patch, > LUCENE-9031.patch, LUCENE-9031.patch, LUCENE-9031.patch > > Time Spent: 1h 50m > Remaining Estimate: 0h > > When UnifiedHighlighter highlights Interval Query it encounters > UnsupportedOperationException. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13870) eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
[
https://issues.apache.org/jira/browse/SOLR-13870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972606#comment-16972606
]
ASF subversion and git services commented on SOLR-13870:
Commit 367fe2f341cca18fb3d22e203b8cd8a91dd36948 in lucene-solr's branch
refs/heads/master from Chris M. Hostetter
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=367fe2f ]
SOLR-13870: SolrJmxReporterTest no longer depends on (buggy)
getNextAvailablePort()
> eliminate getNextAvailablePort() dependency in SolrJmxReporterTest
> --
>
> Key: SOLR-13870
> URL: https://issues.apache.org/jira/browse/SOLR-13870
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-13870.patch
>
>
> {{SolrJmxReporterTest}} is currently the only solr test using (the
> problematic) {{SolrTestCaseJ4.getNextAvailablePort}} helper method in order
> to assign a RMI port for running a JMX server.
> This test doesn't actually care that it's using RMI for JMX, it only cares
> that:
> # an MBeanServer is initialized that it can query
> # that MBeanServer can be specified in the plugin configuration for the
> {{SolrJmxReporter}}
> We should cleanup the test initialization so that it no longer depends on RMI
> via the {{serviceUrl}} plugin configuration option, and instead just use the
> {{agentId}} to specify an MBeanServer bootstrapped by the test.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9042) Refactor TopGroups.merge tests
[
https://issues.apache.org/jira/browse/LUCENE-9042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Diego Ceccarelli updated LUCENE-9042:
-
Status: Patch Available (was: Open)
> Refactor TopGroups.merge tests
> --
>
> Key: LUCENE-9042
> URL: https://issues.apache.org/jira/browse/LUCENE-9042
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Diego Ceccarelli
>Priority: Minor
>
> This task proposes a refactoring of the test coverage for the
> {{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
> only 3 main cases.
> 1. Merging to empty TopGroups
> 2. Merging a TopGroups with scores and a TopGroups without scores (currently
> broken because of LUCENE-8996 bug)
> 3. Merging two TopGroups with scores.
> I'm planning to increase the coverage testing also invalid inputs but I would
> do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13911) Support missing() aggregation in JSON facet module
[
https://issues.apache.org/jira/browse/SOLR-13911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972633#comment-16972633
]
Munendra S N commented on SOLR-13911:
-
[^SOLR-13911.patch]
Thanks for the review.
I have uncommented tests related to {{missing}} aggregation from SOLR-11695 and
added documentation for this func
> Support missing() aggregation in JSON facet module
> --
>
> Key: SOLR-13911
> URL: https://issues.apache.org/jira/browse/SOLR-13911
> Project: Solr
> Issue Type: Sub-task
> Components: Facet Module
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-11695.patch, SOLR-13911.patch, SOLR-13911.patch,
> SOLR-13911.patch
>
>
> Add {{missing()}} aggregation in JSON Facet module similar to
> StatsComponent's missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9042) Refactor TopGroups.merge tests
Diego Ceccarelli created LUCENE-9042:
Summary: Refactor TopGroups.merge tests
Key: LUCENE-9042
URL: https://issues.apache.org/jira/browse/LUCENE-9042
Project: Lucene - Core
Issue Type: Improvement
Reporter: Diego Ceccarelli
This task proposes a refactoring of the test coverage for the
{{TopGroups.merge}} method implemented in LUCENE-9010. For now it will cover
only 3 main cases.
1. Merging to empty TopGroups
2. Merging a TopGroups with scores and a TopGroups without scores (currently
broken because of LUCENE-8996 bug)
3. Merging two TopGroups with scores.
I'm planning to increase the coverage testing also invalid inputs but I would
do that in a separate PR to keep the code readable.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8996) maxScore is sometimes missing from distributed grouped responses
[
https://issues.apache.org/jira/browse/LUCENE-8996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972630#comment-16972630
]
Diego Ceccarelli commented on LUCENE-8996:
--
[~cpoerschke] I realised only after that tests were split and merged in
LUCENE-9010 - I opened another patch to refactor the tests LUCENE-9042
> maxScore is sometimes missing from distributed grouped responses
>
>
> Key: LUCENE-8996
> URL: https://issues.apache.org/jira/browse/LUCENE-8996
> Project: Lucene - Core
> Issue Type: Bug
>Affects Versions: 5.3
>Reporter: Julien Massenet
>Priority: Minor
> Attachments: LUCENE-8996.02.patch, LUCENE-8996.03.patch,
> LUCENE-8996.04.patch, LUCENE-8996.patch, LUCENE-8996.patch,
> lucene_6_5-GroupingMaxScore.patch, lucene_solr_5_3-GroupingMaxScore.patch,
> master-GroupingMaxScore.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This issue occurs when using the grouping feature in distributed mode and
> sorting by score.
> Each group's {{docList}} in the response is supposed to contain a
> {{maxScore}} entry that hold the maximum score for that group. Using the
> current releases, it sometimes happens that this piece of information is not
> included:
> {code}
> {
> "responseHeader": {
> "status": 0,
> "QTime": 42,
> "params": {
> "sort": "score desc",
> "fl": "id,score",
> "q": "_text_:\"72\"",
> "group.limit": "2",
> "group.field": "group2",
> "group.sort": "score desc",
> "group": "true",
> "wt": "json",
> "fq": "group2:72 OR group2:45"
> }
> },
> "grouped": {
> "group2": {
> "matches": 567,
> "groups": [
> {
> "groupValue": 72,
> "doclist": {
> "numFound": 562,
> "start": 0,
> "maxScore": 2.0378063,
> "docs": [
> {
> "id": "29!26551",
> "score": 2.0378063
> },
> {
> "id": "78!11462",
> "score": 2.0298104
> }
> ]
> }
> },
> {
> "groupValue": 45,
> "doclist": {
> "numFound": 5,
> "start": 0,
> "docs": [
> {
> "id": "72!8569",
> "score": 1.8988966
> },
> {
> "id": "72!14075",
> "score": 1.5191172
> }
> ]
> }
> }
> ]
> }
> }
> }
> {code}
> Looking into the issue, it comes from the fact that if a shard does not
> contain a document from that group, trying to merge its {{maxScore}} with
> real {{maxScore}} entries from other shards is invalid (it results in NaN).
> I'm attaching a patch containing a fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13911) Support missing() aggregation in JSON facet module
[
https://issues.apache.org/jira/browse/SOLR-13911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Munendra S N updated SOLR-13911:
Attachment: SOLR-13911.patch
> Support missing() aggregation in JSON facet module
> --
>
> Key: SOLR-13911
> URL: https://issues.apache.org/jira/browse/SOLR-13911
> Project: Solr
> Issue Type: Sub-task
> Components: Facet Module
>Reporter: Munendra S N
>Priority: Major
> Attachments: SOLR-11695.patch, SOLR-13911.patch, SOLR-13911.patch,
> SOLR-13911.patch
>
>
> Add {{missing()}} aggregation in JSON Facet module similar to
> StatsComponent's missing
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972688#comment-16972688
]
Yonik Seeley commented on SOLR-13898:
-
Using computeIfAbsent makes total sense for something like UnInvertedField,
which is huge and expensive and you absolutely don't want to ever build a
duplicate because of a race.
With something like the filter cache, the access pattern tends to be much
different and using computeIfAbsent can be optimizing for the rare case and not
the common case (well, the common case of 100% hit rate for the types of
requests that hit the filter cache many times and expect that hit rate to get
good performance... and on a miss, it's quick to compute anyway). All those
cases (variants of faceting, join, etc) are all requesting filters for terms
though, so perhaps that case could be handled differently from more complex
queries.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13904) Make Analytics component aware of timeAllowed
[ https://issues.apache.org/jira/browse/SOLR-13904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972620#comment-16972620 ] Houston Putman commented on SOLR-13904: --- I'll review this, and I think it's a good thing to add. Instead of just stopping all development, let's: # Deprecate the analytics component # Make a migration path to JSON Facets ## Make sure that JSON Facets has all of the necessary functionality (median, unique, percentile, etc.) ## Have analytics queries be converted to json facets, and computed that way. # Remove the Analytics Component backend # Down the road stop supporting analytics component requests all together (Make users switch to json facets themselves) > Make Analytics component aware of timeAllowed > - > > Key: SOLR-13904 > URL: https://issues.apache.org/jira/browse/SOLR-13904 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mikhail Khludnev >Priority: Major > Attachments: SOLR-13904.patch, SOLR-13904.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13924) MoveReplica failures when using HDFS (NullPointerException)
[
https://issues.apache.org/jira/browse/SOLR-13924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972677#comment-16972677
]
Chris M. Hostetter commented on SOLR-13924:
---
Reviewing the logs for {{MoveReplicaHDFSTest}} jenkins failures from the past 7
days shows a similar pattern in every failure...
* Test ultimately fails with...{noformat}
[junit4] FAILURE 19.4s J0 | MoveReplicaHDFSTest.test <<<
[junit4]> Throwable #1: java.lang.AssertionError: expected not same
[junit4]>at
__randomizedtesting.SeedInfo.seed([7631F50DBFD47AFA:FE65CAD711281702]:0)
[junit4]>at
org.apache.solr.cloud.MoveReplicaTest.test(MoveReplicaTest.java:147)
{noformat}
** This assertion comes from a loop checking the async status of a
{{MoveReplica}} replica request...{code}
assertNotSame(rsp.getRequestStatus(), RequestStatusState.FAILED);
{code}
* Looking back in the logs we see...{noformat}
[junit4] 2> 1325710 ERROR
(OverseerThreadFactory-2562-thread-3-processing-n:127.0.0.1:37772_solr)
[n:127.0.0.1:37772_solr c:MoveReplicaHDFSTest_coll_true r:core_node8 ]
o.a.s.c.a.c.O
verseerCollectionMessageHandler Collection: MoveReplicaHDFSTest_coll_true
operation: movereplica failed:java.lang.NullPointerException
[junit4] 2>at
org.apache.solr.cloud.api.collections.MoveReplicaCmd.moveHdfsReplica(MoveReplicaCmd.java:220)
[junit4] 2>at
org.apache.solr.cloud.api.collections.MoveReplicaCmd.moveReplica(MoveReplicaCmd.java:160)
[junit4] 2>at
org.apache.solr.cloud.api.collections.MoveReplicaCmd.call(MoveReplicaCmd.java:70)
[junit4] 2>at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:263)
{noformat}
If we look back at the long term stats, we see a significant uptick in jenkins
failures from this test starting around 2019-10-18. Note the number of jenkins
failures from this test every day in 2019 (skipping days where there were '0'
failures)...
{noformat}
~/jenkins-reports/output/html/reports/archive/daily$ zgrep -c
MoveReplicaHDFSTest 2019-*method-failures.csv.gz | grep -v ':0$'
2019-03-13.method-failures.csv.gz:1
2019-10-18.method-failures.csv.gz:1
2019-10-19.method-failures.csv.gz:1
2019-10-20.method-failures.csv.gz:8
2019-10-21.method-failures.csv.gz:5
2019-10-23.method-failures.csv.gz:1
2019-10-24.method-failures.csv.gz:2
2019-10-25.method-failures.csv.gz:1
2019-10-26.method-failures.csv.gz:1
2019-10-27.method-failures.csv.gz:2
2019-10-28.method-failures.csv.gz:6
2019-10-29.method-failures.csv.gz:4
2019-10-30.method-failures.csv.gz:2
2019-10-31.method-failures.csv.gz:7
2019-11-01.method-failures.csv.gz:3
2019-11-02.method-failures.csv.gz:2
2019-11-03.method-failures.csv.gz:3
2019-11-04.method-failures.csv.gz:2
2019-11-05.method-failures.csv.gz:7
2019-11-06.method-failures.csv.gz:6
2019-11-08.method-failures.csv.gz:6
2019-11-09.method-failures.csv.gz:6
2019-11-10.method-failures.csv.gz:1
2019-11-11.method-failures.csv.gz:9
2019-11-12.method-failures.csv.gz:10
{noformat}
That date corrispond closely to when SOLR-13843 was commited
(86a40c1cd5691ce8c9c233c9a8186a4f50aa4f5f on master)
If I compare {{git co master}}, {{git co
86a40c1cd5691ce8c9c233c9a8186a4f50aa4f5f}} and {{git co
86a40c1cd5691ce8c9c233c9a8186a4f50aa4f5f~1}} I see that beasting this seed just
a few times fails reliably on {{HEAD}}, and @ {{86a40c1cd56}} but passess
reliably one SHA prior to the SOLR-13843 commit
{noformat}
hossman@slate:~/lucene/dev/solr/core [j11] [master] $ ant beast
-Dbeast.iters=10 -Dtestcase=MoveReplicaHDFSTest -Dtests.nightly=true
-Dtests.slow=true -Dtests.seed=7631F50DBFD47AFA
...
[beaster] Tests with failures [seed: 7631F50DBFD47AFA]:
[beaster] - org.apache.solr.cloud.MoveReplicaHDFSTest.test
{noformat}
{noformat}
hossman@slate:~/lucene/dev/solr/core [j11] [86a40c1cd56] $ ant beast
-Dbeast.iters=10 -Dtestcase=MoveReplicaHDFSTest -Dtests.nightly=true
-Dtests.slow=true -Dtests.seed=7631F50DBFD47AFA
...
[beaster] Tests with failures [seed: 7631F50DBFD47AFA]:
[beaster] - org.apache.solr.cloud.MoveReplicaHDFSTest.test
{noformat}
{noformat}
hossman@slate:~/lucene/dev/solr/core [j11] [63e9bcf5d15] $ ant beast
-Dbeast.iters=10 -Dtestcase=MoveReplicaHDFSTest -Dtests.nightly=true
-Dtests.slow=true -Dtests.seed=7631F50DBFD47AFA
...
[beaster] Beast round 10 results:
/home/hossman/lucene/dev/solr/build/solr-core/test/10
[beaster] Beasting finished Successfully.
{noformat}
> MoveReplica failures when using HDFS (NullPointerException)
> ---
>
> Key: SOLR-13924
> URL: https://issues.apache.org/jira/browse/SOLR-13924
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972678#comment-16972678
]
Andrzej Bialecki commented on SOLR-13898:
-
Updated patch:
* a few bugfixes
* use prescreen {{get(key)}} first in legacy cache implementations.
I also added {{TestSolrCachePerf}} test that tries to expose the difference in
hit ratio between get / put and computeIfAbsent in a highly concurrent use.
Basically the test attempts to create many racing put / get (or
computeIfAbsent) accesses to show that the hit ratio is lower when using get /
put than when using computeIfAbsent. The test passes, ie. it shows that
computeIfAbsent results in a higher hit ratio, but I'm not 100% sure it's the
best way to prove this ;)
In particular, the timings are a bit unexpected (I didn't assert anything yet
about the timings in the test, because the results were puzzling). The test
measures total elapsed time for the whole scenario run, for each cache
implementation, and calculates an average across 100 runs. For {{LFUCache}} and
{{LRUCache}} it actually takes longer when {{computeIfAbsent}} is used. I
suspect it has something to do with excessive synchronization in each impl. but
I'm worried that maybe something is wrong with the testing method (or with the
computeIfAbsent implementation in these caches).
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972682#comment-16972682
]
Ben Manes commented on SOLR-13898:
--
Since {{computeIfAbsent}} is complex and so are the LRU & LFU caches, I
wouldn't have felt confident enough to add an implementation. Of course I am
less familiar with Solr's code, so Instead I would have used a racy version
(ala {{ConcurrentMap}}) rather than an atomic version. Then it is just
refactoring of code structure for taking advantage of atomicity if supported by
the {{SolrCache}}. Since those versions would go away in 9.x and no one
complained, it seems okay to be conservative. But if you're comfortable, it is
great to do for current users.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13888) SolrCloud 2
[ https://issues.apache.org/jira/browse/SOLR-13888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972555#comment-16972555 ] Mark Miller commented on SOLR-13888: The guy that ran the first company I worked for - conservative company, conservative guy, a few months in he tells someone "this kid is going to be superstar or fired". Haha. Any bets on the outcome this time? > SolrCloud 2 > --- > > Key: SOLR-13888 > URL: https://issues.apache.org/jira/browse/SOLR-13888 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Attachments: solrscreen.png > > > As devs discuss dropping the SolrCloud name on the dev list, here is an issue > titled SolrCloud 2. > A couple times now I've pulled on the sweater thread that is our broken > tests. It leads to one place - SolrCloud is sick and devs are adding spotty > code on top of it at a rate that will lead to the system falling in on > itself. As it is, it's a very slow, very inefficient, very unreliable, very > buggy system. > This is not why I am here. This is the opposite of why I am here. > So please, let's stop. We can't build on that thing as it is. > > I need some time, I lost a lot of work at one point, the scope has expanded > since I realized how problematic some things really are, but I have an > alternative path that is not so duct tape and straw. As the building climbs, > that foundation is going to kill us all. > > This i not about an architecture change - the architecture is fine. The > implementation is broken and getting worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13888) SolrCloud 2
[ https://issues.apache.org/jira/browse/SOLR-13888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972579#comment-16972579 ] Mark Miller commented on SOLR-13888: My new teamates at the time? Damn, I think one of them liked me, I still like him, but the rest? Who the hell are you, what is your problem, we now dislike, I was friggen coasting quite happily here. > SolrCloud 2 > --- > > Key: SOLR-13888 > URL: https://issues.apache.org/jira/browse/SOLR-13888 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Attachments: solrscreen.png > > > As devs discuss dropping the SolrCloud name on the dev list, here is an issue > titled SolrCloud 2. > A couple times now I've pulled on the sweater thread that is our broken > tests. It leads to one place - SolrCloud is sick and devs are adding spotty > code on top of it at a rate that will lead to the system falling in on > itself. As it is, it's a very slow, very inefficient, very unreliable, very > buggy system. > This is not why I am here. This is the opposite of why I am here. > So please, let's stop. We can't build on that thing as it is. > > I need some time, I lost a lot of work at one point, the scope has expanded > since I realized how problematic some things really are, but I have an > alternative path that is not so duct tape and straw. As the building climbs, > that foundation is going to kill us all. > > This i not about an architecture change - the architecture is fine. The > implementation is broken and getting worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13888) SolrCloud 2
[ https://issues.apache.org/jira/browse/SOLR-13888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972428#comment-16972428 ] Mark Miller commented on SOLR-13888: "In literature, the *jester* is symbolic of common sense and of honesty" Damn, actually that talented twitter troll was onto something. > SolrCloud 2 > --- > > Key: SOLR-13888 > URL: https://issues.apache.org/jira/browse/SOLR-13888 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Attachments: solrscreen.png > > > As devs discuss dropping the SolrCloud name on the dev list, here is an issue > titled SolrCloud 2. > A couple times now I've pulled on the sweater thread that is our broken > tests. It leads to one place - SolrCloud is sick and devs are adding spotty > code on top of it at a rate that will lead to the system falling in on > itself. As it is, it's a very slow, very inefficient, very unreliable, very > buggy system. > This is not why I am here. This is the opposite of why I am here. > So please, let's stop. We can't build on that thing as it is. > > I need some time, I lost a lot of work at one point, the scope has expanded > since I realized how problematic some things really are, but I have an > alternative path that is not so duct tape and straw. As the building climbs, > that foundation is going to kill us all. > > This i not about an architecture change - the architecture is fine. The > implementation is broken and getting worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-8920) Reduce size of FSTs due to use of direct-addressing encoding
[
https://issues.apache.org/jira/browse/LUCENE-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972414#comment-16972414
]
Michael Sokolov edited comment on LUCENE-8920 at 11/12/19 1:53 PM:
---
+1 for merging, and handling {{cachedRootArcs}} separately
was (Author: sokolov):
+1 for merging, and handling {cachedRootArcs} separately
> Reduce size of FSTs due to use of direct-addressing encoding
> -
>
> Key: LUCENE-8920
> URL: https://issues.apache.org/jira/browse/LUCENE-8920
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael Sokolov
>Priority: Minor
> Attachments: TestTermsDictRamBytesUsed.java
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Some data can lead to worst-case ~4x RAM usage due to this optimization.
> Several ideas were suggested to combat this on the mailing list:
> bq. I think we can improve thesituation here by tracking, per-FST instance,
> the size increase we're seeing while building (or perhaps do a preliminary
> pass before building) in order to decide whether to apply the encoding.
> bq. we could also make the encoding a bit more efficient. For instance I
> noticed that arc metadata is pretty large in some cases (in the 10-20 bytes)
> which make gaps very costly. Associating each label with a dense id and
> having an intermediate lookup, ie. lookup label -> id and then id->arc offset
> instead of doing label->arc directly could save a lot of space in some cases?
> Also it seems that we are repeating the label in the arc metadata when
> array-with-gaps is used, even though it shouldn't be necessary since the
> label is implicit from the address?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8920) Reduce size of FSTs due to use of direct-addressing encoding
[
https://issues.apache.org/jira/browse/LUCENE-8920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972414#comment-16972414
]
Michael Sokolov commented on LUCENE-8920:
-
+1 for merging, and handling {cachedRootArcs} separately
> Reduce size of FSTs due to use of direct-addressing encoding
> -
>
> Key: LUCENE-8920
> URL: https://issues.apache.org/jira/browse/LUCENE-8920
> Project: Lucene - Core
> Issue Type: Improvement
>Reporter: Michael Sokolov
>Priority: Minor
> Attachments: TestTermsDictRamBytesUsed.java
>
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> Some data can lead to worst-case ~4x RAM usage due to this optimization.
> Several ideas were suggested to combat this on the mailing list:
> bq. I think we can improve thesituation here by tracking, per-FST instance,
> the size increase we're seeing while building (or perhaps do a preliminary
> pass before building) in order to decide whether to apply the encoding.
> bq. we could also make the encoding a bit more efficient. For instance I
> noticed that arc metadata is pretty large in some cases (in the 10-20 bytes)
> which make gaps very costly. Associating each label with a dense id and
> having an intermediate lookup, ie. lookup label -> id and then id->arc offset
> instead of doing label->arc directly could save a lot of space in some cases?
> Also it seems that we are repeating the label in the arc metadata when
> array-with-gaps is used, even though it shouldn't be necessary since the
> label is implicit from the address?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13888) SolrCloud 2
[ https://issues.apache.org/jira/browse/SOLR-13888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972569#comment-16972569 ] Mark Miller commented on SOLR-13888: You know what the problem was that time? I was working on garbage, so I rewrote the system with Lucene. Which brought me here. And then money shut me up, cause I'm lazy and have people to support. > SolrCloud 2 > --- > > Key: SOLR-13888 > URL: https://issues.apache.org/jira/browse/SOLR-13888 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Attachments: solrscreen.png > > > As devs discuss dropping the SolrCloud name on the dev list, here is an issue > titled SolrCloud 2. > A couple times now I've pulled on the sweater thread that is our broken > tests. It leads to one place - SolrCloud is sick and devs are adding spotty > code on top of it at a rate that will lead to the system falling in on > itself. As it is, it's a very slow, very inefficient, very unreliable, very > buggy system. > This is not why I am here. This is the opposite of why I am here. > So please, let's stop. We can't build on that thing as it is. > > I need some time, I lost a lot of work at one point, the scope has expanded > since I realized how problematic some things really are, but I have an > alternative path that is not so duct tape and straw. As the building climbs, > that foundation is going to kill us all. > > This i not about an architecture change - the architecture is fine. The > implementation is broken and getting worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-13923) Test target (task?) should fail when no tests run
Michael Sokolov created SOLR-13923:
--
Summary: Test target (task?) should fail when no tests run
Key: SOLR-13923
URL: https://issues.apache.org/jira/browse/SOLR-13923
Project: Solr
Issue Type: Sub-task
Security Level: Public (Default Security Level. Issues are Public)
Components: Build
Reporter: Michael Sokolov
With the ant build if you try to test a nonexistent test case or method
({{-Dtestcase=NoSuchThing}}, the build will fail; this is pretty helpful if you
make a lot of typos or forget the names of things. According to [~dweiss] we
can get this behavior in gradle by listening to the test results and failing if
no tests ran.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-13923) Test target (task?) should fail when no tests run
[
https://issues.apache.org/jira/browse/SOLR-13923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Sokolov updated SOLR-13923:
---
Parent: (was: SOLR-13914)
Issue Type: Task (was: Sub-task)
> Test target (task?) should fail when no tests run
> -
>
> Key: SOLR-13923
> URL: https://issues.apache.org/jira/browse/SOLR-13923
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: Build
>Reporter: Michael Sokolov
>Priority: Minor
>
> With the ant build if you try to test a nonexistent test case or method
> ({{-Dtestcase=NoSuchThing}}, the build will fail; this is pretty helpful if
> you make a lot of typos or forget the names of things. According to [~dweiss]
> we can get this behavior in gradle by listening to the test results and
> failing if no tests ran.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13888) SolrCloud 2
[ https://issues.apache.org/jira/browse/SOLR-13888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972507#comment-16972507 ] Mark Miller commented on SOLR-13888: Now there is something wrong wit my dog, life won't quit it's been after me since I started. Look guys, from day one I've thought I was idiot. That ends up creating a problem when I finally decide I know more than something about you. Problems though, they are meals. I've never not had them. So it's okay. This is easy :) If I can do it anyone can. But it is time. > SolrCloud 2 > --- > > Key: SOLR-13888 > URL: https://issues.apache.org/jira/browse/SOLR-13888 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Major > Attachments: solrscreen.png > > > As devs discuss dropping the SolrCloud name on the dev list, here is an issue > titled SolrCloud 2. > A couple times now I've pulled on the sweater thread that is our broken > tests. It leads to one place - SolrCloud is sick and devs are adding spotty > code on top of it at a rate that will lead to the system falling in on > itself. As it is, it's a very slow, very inefficient, very unreliable, very > buggy system. > This is not why I am here. This is the opposite of why I am here. > So please, let's stop. We can't build on that thing as it is. > > I need some time, I lost a lot of work at one point, the scope has expanded > since I realized how problematic some things really are, but I have an > alternative path that is not so duct tape and straw. As the building climbs, > that foundation is going to kill us all. > > This i not about an architecture change - the architecture is fine. The > implementation is broken and getting worse. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13922) Gradle build output folder and relative adoc includes
[
https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss resolved SOLR-13922.
Resolution: Fixed
{code:java}
.\gradlew -p solr\solr-ref-guide buildSite{code}
Compiles for me cleanly now (no warnings).
There are link validation errors when I run:
{code:java}
.\gradlew -p solr\solr-ref-guide bareBonesHtmlValidation {code}
but these seem unrelated.
> Gradle build output folder and relative adoc includes
> -
>
> Key: SOLR-13922
> URL: https://issues.apache.org/jira/browse/SOLR-13922
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Erick Erickson
>Assignee: Dawid Weiss
>Priority: Major
>
> Raising and assigning as per Dawid's comment on the dev list
> In the gradle_7 branch the HTML files were generated in
> ./solr/solr-ref-guide/build, but now they are in the top-level ./build
> directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13922) Gradle build output folder and relative adoc includes
[ https://issues.apache.org/jira/browse/SOLR-13922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972819#comment-16972819 ] ASF subversion and git services commented on SOLR-13922: Commit b075acbd20f04c93e1273c16a20d2dd7326d3993 in lucene-solr's branch refs/heads/jira/SOLR-13452_gradle_8 from Dawid Weiss [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=b075acb ] SOLR-13922: Gradle build output folder and relative adoc includes > Gradle build output folder and relative adoc includes > - > > Key: SOLR-13922 > URL: https://issues.apache.org/jira/browse/SOLR-13922 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Erick Erickson >Assignee: Dawid Weiss >Priority: Major > > Raising and assigning as per Dawid's comment on the dev list > In the gradle_7 branch the HTML files were generated in > ./solr/solr-ref-guide/build, but now they are in the top-level ./build > directory. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13924) MoveReplica failures when using HDFS (NullPointerException)
[
https://issues.apache.org/jira/browse/SOLR-13924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972864#comment-16972864
]
Shalin Shekhar Mangar commented on SOLR-13924:
--
The changes to the MoveReplicaTest in SOLR-13843 started adding either a TLOG
or a PULL replica to the collection in addition to the NRT replica. This
exposed a genuine bug. Move replica operation does not work for PULL replicas
stored on HDFS. We just didn't know so far because there were no tests for it.
Basically move replica operation for HDFS assumes that each replica has an
update log directory containing the tlog files but the PULL type replicas do
not have them. This causes the NPE.
> MoveReplica failures when using HDFS (NullPointerException)
> ---
>
> Key: SOLR-13924
> URL: https://issues.apache.org/jira/browse/SOLR-13924
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Chris M. Hostetter
>Assignee: Shalin Shekhar Mangar
>Priority: Major
>
> Based on recent jenkins test failures, it appears that attemping to use the
> "MoveReplica" command on HDFS has a high chance of failure due to an
> underlying NPE.
> I'm not sure if this bug *only* affects HDFS, or if it's just more likly to
> occur when using HDFS due to some timing quirks.
> It's also possible that the bug impacts non-HDFS users just as much as HDFS
> users, but only manifests in our tests due to some quick of our
> {{cloud-hdfs}} test configs.
> The problem appears to be new in 8.3 as a result of changes made in SOLR-13843
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13864) MathExpressionTest non-reproducible failures due to assertions of non-absolutes and randomization beyond test seed
[
https://issues.apache.org/jira/browse/SOLR-13864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972916#comment-16972916
]
Joel Bernstein commented on SOLR-13864:
---
[~hossman], I'll fix the others as well.
> MathExpressionTest non-reproducible failures due to assertions of
> non-absolutes and randomization beyond test seed
> --
>
> Key: SOLR-13864
> URL: https://issues.apache.org/jira/browse/SOLR-13864
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Joel Bernstein
>Priority: Major
> Attachments: apache_Lucene-Solr-BadApples-Tests-master_531.log.txt
>
>
> We're seeing a a fairly steady trickle of MathExpressionTest from various
> jenkins boxes going back quite a while ... mostly from testGammaDistribution,
> but other tests pop up now and then.
> the crux of the problem with this test seems to break down into 2 categories:
> # tests that make assumptions about the relative values that will come out
> of taking samples from different random distributions that aren't garunteed
> to be true
> ** ie: comparing 2 random samples from 2 diff shaped gamma distributions and
> expecting one to always be strictly greater then the other. I'm not a stats
> guy, but my naive understanding is that on the low end some of these shapes
> may cross over, so every possible random sample from one shape is not
> garunteed to be less then every ossible random sample from a diff shape
> # the code being tested does it's own randomization outside of the crontrol
> of the test framework (or test client)
> ** this causes the seeds to not reproduce
>
> Tests should not be making assertions about random data that aren't 100%
> garunteed to be true in all cases (ie: {{random().nextInt(5) < (5.0D +
> (double) random().nextInt(5))}} is one thing, {{random().nextInt(5) <
> (4.9D + (double) random().nextInt(5))}} is a diff story.
> Randomized behavior in solr (non-test) code should ideally have some way for
> being controlled by the client/tests ... either via a request param used to
> initialize any new Random instances, or for example the use of the
> "tests.seed" property in various places in the code to try and provide some
> reproducibility even when the external solr client isn't even aware of
> randomization being a factor in the behavior of the code.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345492886
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items = new
ObjectMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = new ObjectMapper().readValue(existingRepositoriesJson,
List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json", new
ObjectMapper().writeValueAsString(repos).getBytes("UTF-8"),
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345493353
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/RepositoryManager.java
##
@@ -0,0 +1,328 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.io.UnsupportedEncodingException;
+import java.lang.invoke.MethodHandles;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.io.IOUtils;
+import org.apache.lucene.util.Version;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.request.beans.Package;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Artifact;
+import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease;
+import org.apache.solr.pkg.PackageAPI;
+import org.apache.zookeeper.CreateMode;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+
+/**
+ * Handles most of the management of repositories and packages present in
external repositories.
+ */
+public class RepositoryManager {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ final private PackageManager packageManager;
+
+ public static final String systemVersion = Version.LATEST.toString();
+
+ final HttpSolrClient solrClient;
+
+ public RepositoryManager(HttpSolrClient solrClient, PackageManager
packageManager) {
+this.packageManager = packageManager;
+this.solrClient = solrClient;
+ }
+
+ public List getPackages() {
+List list = new ArrayList<>(getPackagesMap().values());
+Collections.sort(list);
+return list;
+ }
+
+ /**
+ * Get a map of package name to {@link SolrPackage} objects
+ */
+ public Map getPackagesMap() {
+Map packagesMap = new HashMap<>();
+for (PackageRepository repository: getRepositories()) {
+ packagesMap.putAll(repository.getPackages());
+}
+
+return packagesMap;
+ }
+
+ /**
+ * List of added repositories
+ */
+ public List getRepositories() {
+// TODO: Instead of fetching again and again, we should look for caching
this
+PackageRepository items[];
+try {
+ items = new
ObjectMapper().readValue(getRepositoriesJson(packageManager.zkClient),
DefaultPackageRepository[].class);
+} catch (IOException | KeeperException | InterruptedException e) {
+ throw new SolrException(ErrorCode.SERVER_ERROR, e);
+}
+List repositories = Arrays.asList(items);
+
+for (PackageRepository updateRepository: repositories) {
+ updateRepository.refresh();
+}
+
+return repositories;
+ }
+
+ /**
+ * Add a repository to Solr
+ */
+ public void addRepository(String name, String uri) throws KeeperException,
InterruptedException, MalformedURLException, IOException {
+String existingRepositoriesJson =
getRepositoriesJson(packageManager.zkClient);
+log.info(existingRepositoriesJson);
+
+List repos = new ObjectMapper().readValue(existingRepositoriesJson,
List.class);
+repos.add(new DefaultPackageRepository(name, uri));
+if (packageManager.zkClient.exists("/repositories.json", true) == false) {
+ packageManager.zkClient.create("/repositories.json", new
ObjectMapper().writeValueAsString(repos).getBytes("UTF-8"),
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345492704
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -0,0 +1,212 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.net.MalformedURLException;
+import java.nio.ByteBuffer;
+import java.nio.file.Path;
+import java.util.List;
+import java.util.Map;
+import java.util.zip.ZipEntry;
+import java.util.zip.ZipFile;
+
+import org.apache.commons.io.IOUtils;
+import org.apache.http.client.HttpClient;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.lucene.util.SuppressForbidden;
+import org.apache.solr.client.solrj.SolrClient;
+import org.apache.solr.client.solrj.SolrRequest;
+import org.apache.solr.client.solrj.SolrServerException;
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.client.solrj.request.V2Request;
+import org.apache.solr.client.solrj.response.V2Response;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.params.CommonParams;
+import org.apache.solr.common.params.ModifiableSolrParams;
+import org.apache.solr.core.BlobRepository;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.github.zafarkhaja.semver.Version;
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.Configuration;
+import com.jayway.jsonpath.spi.json.JacksonJsonProvider;
+import com.jayway.jsonpath.spi.json.JsonProvider;
+import com.jayway.jsonpath.spi.mapper.JacksonMappingProvider;
+import com.jayway.jsonpath.spi.mapper.MappingProvider;
+
+public class PackageUtils {
+
+ public static Configuration jsonPathConfiguration() {
+MappingProvider provider = new JacksonMappingProvider();
+JsonProvider jsonProvider = new JacksonJsonProvider();
+Configuration c =
Configuration.builder().jsonProvider(jsonProvider).mappingProvider(provider).options(com.jayway.jsonpath.Option.REQUIRE_PROPERTIES).build();
+return c;
+ }
+
+ /**
+ * Uploads a file to the package store / file store of Solr.
+ *
+ * @param client A Solr client
+ * @param buffer File contents
+ * @param name Name of the file as it will appear in the file store (can be
hierarchical)
+ * @param sig Signature digest (public key should be separately uploaded to
ZK)
+ */
+ public static void postFile(SolrClient client, ByteBuffer buffer, String
name, String sig)
+ throws SolrServerException, IOException {
+String resource = "/api/cluster/files" + name;
+ModifiableSolrParams params = new ModifiableSolrParams();
+if (sig != null) {
+ params.add("sig", sig);
+}
+V2Response rsp = new V2Request.Builder(resource)
+.withMethod(SolrRequest.METHOD.PUT)
+.withPayload(buffer)
+.forceV2(true)
Review comment:
I tried removing this, but tests failed (test doesn't directly called this;
it is CLI calling Solr). Is there any downside to keeping it in?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345492421
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageRepository.java
##
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.packagemanager;
+
+import java.io.IOException;
+import java.nio.file.Path;
+import java.util.Map;
+
+import org.apache.solr.common.SolrException;
+
+import com.fasterxml.jackson.annotation.JsonProperty;
+
+/**
+ * Abstract class for a repository, holding {@link SolrPackage} items.
+ */
+public abstract class PackageRepository {
+
+ @JsonProperty("name")
Review comment:
Done, thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13898) Non-atomic use of SolrCache get / put
[
https://issues.apache.org/jira/browse/SOLR-13898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973008#comment-16973008
]
David Smiley commented on SOLR-13898:
-
Indeed, {{org.apache.solr.search.SolrDocumentFetcher#doc(int,
org.apache.lucene.index.StoredFieldVisitor)}} does not populate the cache but
it does try to fetch from the cache. The highlighter, which uses this, is run
after QueryComponent primes the cache in doPrefetch calling
{{org.apache.solr.util.SolrPluginUtils#optimizePreFetchDocs}} (see it's nice
javadocs). That code was added in 2006. All this said... it'd be nice if the
cache were populated if for some reason it wasn't already populated. Why not.
> Non-atomic use of SolrCache get / put
> -
>
> Key: SOLR-13898
> URL: https://issues.apache.org/jira/browse/SOLR-13898
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.3
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13898.patch, SOLR-13898.patch, SOLR-13898.patch
>
>
> As pointed out by [~ben.manes] in SOLR-13817 Solr code base in many key
> places uses a similar pattern of non-atomic get / put calls to SolrCache-s.
> In multi-threaded environment this leads to cache misses and additional
> unnecessary computations when competing threads both discover a missing
> value, non-atomically compute it and update the cache.
> Some of these places are known performance bottlenecks where efficient
> caching is especially important, such as {{SolrIndexSearcher}},
> {{SolrDocumentFetcher}}, {{UninvertedField}} and join queries .
> I propose to add {{SolrCache.computeIfAbsent(key, mappingFunction)}} that
> will atomically retrieve existing values or compute and update the cache.
> This will require also changing how the {{SolrCache.get(...)}} is used in
> many components.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9018) Separator for ConcatenateGraphFilterFactory
[ https://issues.apache.org/jira/browse/LUCENE-9018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Smiley updated LUCENE-9018: - Status: Patch Available (was: Open) > Separator for ConcatenateGraphFilterFactory > --- > > Key: LUCENE-9018 > URL: https://issues.apache.org/jira/browse/LUCENE-9018 > Project: Lucene - Core > Issue Type: Improvement > Components: modules/analysis >Reporter: Stanislav Mikulchik >Assignee: David Smiley >Priority: Minor > Attachments: LUCENE-9018.patch, LUCENE-9018.patch, LUCENE-9018.patch > > > I would like to have an option to choose a separator to use for token > concatenation. Currently ConcatenateGraphFilterFactory can use only "\u001F" > symbol. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345567216 ## File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackageRepository.java ## @@ -0,0 +1,130 @@ +package org.apache.solr.packagemanager; + +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.Reader; +import java.net.MalformedURLException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import org.apache.commons.io.FileUtils; +import org.apache.solr.common.SolrException; +import org.apache.solr.common.SolrException.ErrorCode; +import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; Review comment: This mistake is extremely easy to make. Perhaps your new json annotation processory thingy should detect `com.fasterxml.jackson.annotation.JsonProperty` and fail? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345567316 ## File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackageRepository.java ## @@ -0,0 +1,130 @@ +package org.apache.solr.packagemanager; + +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.Reader; +import java.net.MalformedURLException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import org.apache.commons.io.FileUtils; +import org.apache.solr.common.SolrException; +import org.apache.solr.common.SolrException.ErrorCode; +import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; Review comment: Or configure forbiddenAPI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13841) Provide mappings for jackson annotation @JsonProperty to use Jackson deserializer
[ https://issues.apache.org/jira/browse/SOLR-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973012#comment-16973012 ] ASF subversion and git services commented on SOLR-13841: Commit 11fd8d630a5e92ab286b378e69e85fa23fddafb1 in lucene-solr's branch refs/heads/master from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=11fd8d6 ] SOLR-13841: Added 'JsonProperty' to forbidden APIs > Provide mappings for jackson annotation @JsonProperty to use Jackson > deserializer > - > > Key: SOLR-13841 > URL: https://issues.apache.org/jira/browse/SOLR-13841 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We can start using annotations in SolrJ to minimize the amount of code we > write & improve readability. Jackson is a widely used library and everyone is > already familiar with it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI) URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345570104 ## File path: solr/core/src/java/org/apache/solr/packagemanager/SolrPackageRepository.java ## @@ -0,0 +1,130 @@ +package org.apache.solr.packagemanager; + +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.Reader; +import java.net.MalformedURLException; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; + +import org.apache.commons.io.FileUtils; +import org.apache.solr.common.SolrException; +import org.apache.solr.common.SolrException.ErrorCode; +import org.apache.solr.packagemanager.SolrPackage.SolrPackageRelease; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import com.fasterxml.jackson.annotation.JsonIgnore; +import com.fasterxml.jackson.annotation.JsonProperty; Review comment: +1 to forbidden API. I had suggested (privately) to Noble to put jackson's JsonProperty to forbidden APIs and he agreed. Also, I think JsonIgnore should be added to Solr's annotations, instead of my trying to use jackson's version. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345571223
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -90,52 +88,123 @@ public static void postFile(SolrClient client, ByteBuffer
buffer, String name, S
}
}
- public static String getStringFromStream(String url) {
-return get(url);
+ /**
+ * Download JSON from the url and deserialize into klass.
+ */
+ public static T getJson(HttpClient client, String url, Class klass) {
+try {
+ return new ObjectMapper().readValue(getJson(client, url), klass);
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
}
- public static String get(String url) {
-try (CloseableHttpClient client =
PackageUtils.createTrustAllHttpClientBuilder()) {
- HttpGet httpGet = new HttpGet(url);
- httpGet.setHeader("Content-type", "application/json");
-
- CloseableHttpResponse response = client.execute(httpGet);
-
- try {
-HttpEntity rspEntity = response.getEntity();
-if (rspEntity != null) {
- InputStream is = rspEntity.getContent();
- StringWriter writer = new StringWriter();
- IOUtils.copy(is, writer, "UTF-8");
- String results = writer.toString();
-
- return(results);
-}
- } catch (IOException e) {
-e.printStackTrace();
+ /**
+ * Search through the list of jar files for a given file. Returns string of
+ * the file contents or null if file wasn't found. This is suitable for
looking
+ * for manifest or property files within pre-downloaded jar files.
+ */
+ public static String getFileFromArtifacts(List jars, String filename) {
+for (Path jarfile: jars) {
+ try (ZipFile zipFile = new ZipFile(jarfile.toFile())) {
+ZipEntry entry = zipFile.getEntry(filename);
+if (entry == null) continue;
+return IOUtils.toString(zipFile.getInputStream(entry));
+ } catch (Exception ex) {
+throw new SolrException(ErrorCode.BAD_REQUEST, ex);
}
-} catch (Exception e1) {
- throw new RuntimeException(e1);
}
return null;
}
- public static CloseableHttpClient createTrustAllHttpClientBuilder() throws
Exception {
-SSLContextBuilder builder = new SSLContextBuilder();
-builder.loadTrustMaterial(null, (chain, authType) -> true);
-SSLConnectionSocketFactory sslsf = new
-SSLConnectionSocketFactory(builder.build(),
NoopHostnameVerifier.INSTANCE);
-return HttpClients.custom().setSSLSocketFactory(sslsf).build();
+ /**
+ * Returns JSON string from a given URL
+ */
+ public static String getJson(HttpClient client, String url) {
Review comment:
Rename: getJsonStringFromUrl
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345570908
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -90,52 +88,123 @@ public static void postFile(SolrClient client, ByteBuffer
buffer, String name, S
}
}
- public static String getStringFromStream(String url) {
-return get(url);
+ /**
+ * Download JSON from the url and deserialize into klass.
+ */
+ public static T getJson(HttpClient client, String url, Class klass) {
+try {
+ return new ObjectMapper().readValue(getJson(client, url), klass);
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
}
- public static String get(String url) {
-try (CloseableHttpClient client =
PackageUtils.createTrustAllHttpClientBuilder()) {
- HttpGet httpGet = new HttpGet(url);
- httpGet.setHeader("Content-type", "application/json");
-
- CloseableHttpResponse response = client.execute(httpGet);
-
- try {
-HttpEntity rspEntity = response.getEntity();
-if (rspEntity != null) {
- InputStream is = rspEntity.getContent();
- StringWriter writer = new StringWriter();
- IOUtils.copy(is, writer, "UTF-8");
- String results = writer.toString();
-
- return(results);
-}
- } catch (IOException e) {
-e.printStackTrace();
+ /**
+ * Search through the list of jar files for a given file. Returns string of
+ * the file contents or null if file wasn't found. This is suitable for
looking
+ * for manifest or property files within pre-downloaded jar files.
+ */
+ public static String getFileFromArtifacts(List jars, String filename) {
Review comment:
I suggest appending "AsString" to the name. Maybe replace "Artifacts" with
"Jars" at your discretion.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345570063
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##
@@ -0,0 +1,406 @@
+package org.apache.solr.packagemanager;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.packagemanager.SolrPackage.Command;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+import org.apache.solr.packagemanager.SolrPackage.Plugin;
+import org.apache.solr.util.SolrCLI;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.PathNotFoundException;
+
+/**
+ * Handles most of the management of packages that are already installed in
Solr.
+ */
+public class PackageManager implements Closeable {
+
+ final String solrBaseUrl;
+ final HttpSolrClient solrClient;
+ final SolrZkClient zkClient;
+
+ private Map> packages = null;
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+
+ public PackageManager(HttpSolrClient solrClient, String solrBaseUrl, String
zkHost) {
+this.solrBaseUrl = solrBaseUrl;
+this.solrClient = solrClient;
+this.zkClient = new SolrZkClient(zkHost, 3);
+log.info("Done initializing a zkClient instance...");
+ }
+
+ @Override
+ public void close() throws IOException {
+if (zkClient != null) {
+ zkClient.close();
+}
+ }
+
+ public List fetchInstalledPackageInstances() throws
SolrException {
+log.info("Getting packages from packages.json...");
+List ret = new ArrayList();
+packages = new HashMap>();
+try {
+ Map packagesZnodeMap = null;
+
+ if (zkClient.exists("/packages.json", true) == true) {
+packagesZnodeMap = (Map)new ObjectMapper().readValue(
+new String(zkClient.getData("/packages.json", null, null, true),
"UTF-8"), Map.class).get("packages");
+for (Object packageName: packagesZnodeMap.keySet()) {
+ List pkg = (List)packagesZnodeMap.get(packageName);
+ for (Map pkgVersion: (List)pkg) {
+Manifest manifest = PackageUtils.fetchManifest(solrClient,
solrBaseUrl, pkgVersion.get("manifest").toString(),
pkgVersion.get("manifestSHA512").toString());
+List solrplugins = manifest.plugins;
+SolrPackageInstance pkgInstance = new
SolrPackageInstance(packageName.toString(), null,
+pkgVersion.get("version").toString(), manifest, solrplugins,
manifest.parameterDefaults);
+List list =
packages.containsKey(packageName)? packages.get(packageName): new
ArrayList();
+list.add(pkgInstance);
+packages.put(packageName.toString(), list);
+ret.add(pkgInstance);
+ }
+}
+ }
+} catch (Exception e) {
+ throw new SolrException(ErrorCode.BAD_REQUEST, e);
+}
+log.info("Got packages: "+ret);
+return ret;
+ }
+
+ public Map getPackagesDeployed(String
collection) {
+String paramsJson = PackageUtils.getJson(solrClient.getHttpClient(),
solrBaseUrl+"/api/collections/"+collection+"/config/params/PKG_VERSIONS?omitHeader=true");
+Map packages = null;
+try {
+ packages = JsonPath.parse(paramsJson,
PackageUtils.jsonPathConfiguration())
+ .read("$['response'].['params'].['PKG_VERSIONS'])", Map.class);
+} catch (PathNotFoundException ex) {
+ // Don't worry if PKG_VERSION wasn't found. It just means this
collection was never touched by the package manager.
+}
+
+if (packages == null) return Collections.emptyMap();
+Map ret = new HashMap();
+for (String packageName: packages.keySet()) {
+ if (Strings.isNullOrEmpty(packageName) == false) { // There can be an
empty key, storing the version here
+ret.put(packageName, getPackageInstance(packageName,
packages.get(packageName)));
+ }
+}
+return ret;
+ }
+
+ private boolean deployPackage(SolrPackageInstance packageInstance, boolean
pegToLatest, boolean isUpdate, boolean noprompt,
+ List collections, String overrides[]) {
+for (String collection: collections) {
+
+ SolrPackageInstance deployedPackage =
getPackagesDeployed(collection).get(packageInstance.name);
+ if
[GitHub] [lucene-solr] dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
dsmiley commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345570063
##
File path:
solr/core/src/java/org/apache/solr/packagemanager/PackageManager.java
##
@@ -0,0 +1,406 @@
+package org.apache.solr.packagemanager;
+
+import java.io.Closeable;
+import java.io.IOException;
+import java.lang.invoke.MethodHandles;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Scanner;
+
+import org.apache.solr.client.solrj.impl.HttpSolrClient;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.SolrException.ErrorCode;
+import org.apache.solr.common.cloud.SolrZkClient;
+import org.apache.solr.packagemanager.SolrPackage.Command;
+import org.apache.solr.packagemanager.SolrPackage.Manifest;
+import org.apache.solr.packagemanager.SolrPackage.Plugin;
+import org.apache.solr.util.SolrCLI;
+import org.apache.zookeeper.KeeperException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.fasterxml.jackson.databind.ObjectMapper;
+import com.google.common.base.Strings;
+import com.jayway.jsonpath.JsonPath;
+import com.jayway.jsonpath.PathNotFoundException;
+
+/**
+ * Handles most of the management of packages that are already installed in
Solr.
+ */
+public class PackageManager implements Closeable {
+
+ final String solrBaseUrl;
+ final HttpSolrClient solrClient;
+ final SolrZkClient zkClient;
+
+ private Map> packages = null;
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+
+ public PackageManager(HttpSolrClient solrClient, String solrBaseUrl, String
zkHost) {
+this.solrBaseUrl = solrBaseUrl;
+this.solrClient = solrClient;
+this.zkClient = new SolrZkClient(zkHost, 3);
+log.info("Done initializing a zkClient instance...");
+ }
+
+ @Override
+ public void close() throws IOException {
+if (zkClient != null) {
+ zkClient.close();
+}
+ }
+
+ public List fetchInstalledPackageInstances() throws
SolrException {
+log.info("Getting packages from packages.json...");
+List ret = new ArrayList();
+packages = new HashMap>();
+try {
+ Map packagesZnodeMap = null;
+
+ if (zkClient.exists("/packages.json", true) == true) {
+packagesZnodeMap = (Map)new ObjectMapper().readValue(
+new String(zkClient.getData("/packages.json", null, null, true),
"UTF-8"), Map.class).get("packages");
+for (Object packageName: packagesZnodeMap.keySet()) {
+ List pkg = (List)packagesZnodeMap.get(packageName);
+ for (Map pkgVersion: (List)pkg) {
+Manifest manifest = PackageUtils.fetchManifest(solrClient,
solrBaseUrl, pkgVersion.get("manifest").toString(),
pkgVersion.get("manifestSHA512").toString());
+List solrplugins = manifest.plugins;
+SolrPackageInstance pkgInstance = new
SolrPackageInstance(packageName.toString(), null,
+pkgVersion.get("version").toString(), manifest, solrplugins,
manifest.parameterDefaults);
+List list =
packages.containsKey(packageName)? packages.get(packageName): new
ArrayList();
+list.add(pkgInstance);
+packages.put(packageName.toString(), list);
+ret.add(pkgInstance);
+ }
+}
+ }
+} catch (Exception e) {
+ throw new SolrException(ErrorCode.BAD_REQUEST, e);
+}
+log.info("Got packages: "+ret);
+return ret;
+ }
+
+ public Map getPackagesDeployed(String
collection) {
+String paramsJson = PackageUtils.getJson(solrClient.getHttpClient(),
solrBaseUrl+"/api/collections/"+collection+"/config/params/PKG_VERSIONS?omitHeader=true");
+Map packages = null;
+try {
+ packages = JsonPath.parse(paramsJson,
PackageUtils.jsonPathConfiguration())
+ .read("$['response'].['params'].['PKG_VERSIONS'])", Map.class);
+} catch (PathNotFoundException ex) {
+ // Don't worry if PKG_VERSION wasn't found. It just means this
collection was never touched by the package manager.
+}
+
+if (packages == null) return Collections.emptyMap();
+Map ret = new HashMap();
+for (String packageName: packages.keySet()) {
+ if (Strings.isNullOrEmpty(packageName) == false) { // There can be an
empty key, storing the version here
+ret.put(packageName, getPackageInstance(packageName,
packages.get(packageName)));
+ }
+}
+return ret;
+ }
+
+ private boolean deployPackage(SolrPackageInstance packageInstance, boolean
pegToLatest, boolean isUpdate, boolean noprompt,
+ List collections, String overrides[]) {
+for (String collection: collections) {
+
+ SolrPackageInstance deployedPackage =
getPackagesDeployed(collection).get(packageInstance.name);
+ if
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345573855
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -90,52 +88,123 @@ public static void postFile(SolrClient client, ByteBuffer
buffer, String name, S
}
}
- public static String getStringFromStream(String url) {
-return get(url);
+ /**
+ * Download JSON from the url and deserialize into klass.
+ */
+ public static T getJson(HttpClient client, String url, Class klass) {
+try {
+ return new ObjectMapper().readValue(getJson(client, url), klass);
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
}
- public static String get(String url) {
-try (CloseableHttpClient client =
PackageUtils.createTrustAllHttpClientBuilder()) {
- HttpGet httpGet = new HttpGet(url);
- httpGet.setHeader("Content-type", "application/json");
-
- CloseableHttpResponse response = client.execute(httpGet);
-
- try {
-HttpEntity rspEntity = response.getEntity();
-if (rspEntity != null) {
- InputStream is = rspEntity.getContent();
- StringWriter writer = new StringWriter();
- IOUtils.copy(is, writer, "UTF-8");
- String results = writer.toString();
-
- return(results);
-}
- } catch (IOException e) {
-e.printStackTrace();
+ /**
+ * Search through the list of jar files for a given file. Returns string of
+ * the file contents or null if file wasn't found. This is suitable for
looking
+ * for manifest or property files within pre-downloaded jar files.
+ */
+ public static String getFileFromArtifacts(List jars, String filename) {
+for (Path jarfile: jars) {
+ try (ZipFile zipFile = new ZipFile(jarfile.toFile())) {
+ZipEntry entry = zipFile.getEntry(filename);
+if (entry == null) continue;
+return IOUtils.toString(zipFile.getInputStream(entry));
+ } catch (Exception ex) {
+throw new SolrException(ErrorCode.BAD_REQUEST, ex);
}
-} catch (Exception e1) {
- throw new RuntimeException(e1);
}
return null;
}
- public static CloseableHttpClient createTrustAllHttpClientBuilder() throws
Exception {
-SSLContextBuilder builder = new SSLContextBuilder();
-builder.loadTrustMaterial(null, (chain, authType) -> true);
-SSLConnectionSocketFactory sslsf = new
-SSLConnectionSocketFactory(builder.build(),
NoopHostnameVerifier.INSTANCE);
-return HttpClients.custom().setSSLSocketFactory(sslsf).build();
+ /**
+ * Returns JSON string from a given URL
+ */
+ public static String getJson(HttpClient client, String url) {
Review comment:
Sounds good, I'll make that change!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] chatman commented on a change in pull request #994: SOLR-13662: Package Manager (CLI)
chatman commented on a change in pull request #994: SOLR-13662: Package Manager
(CLI)
URL: https://github.com/apache/lucene-solr/pull/994#discussion_r345573831
##
File path: solr/core/src/java/org/apache/solr/packagemanager/PackageUtils.java
##
@@ -90,52 +88,123 @@ public static void postFile(SolrClient client, ByteBuffer
buffer, String name, S
}
}
- public static String getStringFromStream(String url) {
-return get(url);
+ /**
+ * Download JSON from the url and deserialize into klass.
+ */
+ public static T getJson(HttpClient client, String url, Class klass) {
+try {
+ return new ObjectMapper().readValue(getJson(client, url), klass);
+} catch (IOException e) {
+ throw new RuntimeException(e);
+}
}
- public static String get(String url) {
-try (CloseableHttpClient client =
PackageUtils.createTrustAllHttpClientBuilder()) {
- HttpGet httpGet = new HttpGet(url);
- httpGet.setHeader("Content-type", "application/json");
-
- CloseableHttpResponse response = client.execute(httpGet);
-
- try {
-HttpEntity rspEntity = response.getEntity();
-if (rspEntity != null) {
- InputStream is = rspEntity.getContent();
- StringWriter writer = new StringWriter();
- IOUtils.copy(is, writer, "UTF-8");
- String results = writer.toString();
-
- return(results);
-}
- } catch (IOException e) {
-e.printStackTrace();
+ /**
+ * Search through the list of jar files for a given file. Returns string of
+ * the file contents or null if file wasn't found. This is suitable for
looking
+ * for manifest or property files within pre-downloaded jar files.
+ */
+ public static String getFileFromArtifacts(List jars, String filename) {
Review comment:
Sounds good, I'll make that change!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13872) Backup can fail to read index files w/NoSuchFileException during merges (SOLR-11616 regression)
[
https://issues.apache.org/jira/browse/SOLR-13872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16972973#comment-16972973
]
Lucene/Solr QA commented on SOLR-13872:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Release audit (RAT) {color} |
{color:green} 1m 5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Check forbidden APIs {color} |
{color:green} 1m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate source patterns {color} |
{color:green} 1m 2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} Validate ref guide {color} |
{color:green} 1m 2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 46m
17s{color} | {color:green} core in the patch passed. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | SOLR-13872 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12985582/SOLR-13872.patch |
| Optional Tests | compile javac unit ratsources checkforbiddenapis
validatesourcepatterns validaterefguide |
| uname | Linux lucene1-us-west 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | ant |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-SOLR-Build/sourcedir/dev-tools/test-patch/lucene-solr-yetus-personality.sh
|
| git revision | master / 5df9a51cbfb |
| ant | version: Apache Ant(TM) version 1.10.5 compiled on March 28 2019 |
| Default Java | LTS |
| Test Results |
https://builds.apache.org/job/PreCommit-SOLR-Build/595/testReport/ |
| modules | C: solr/core solr/solr-ref-guide U: solr |
| Console output |
https://builds.apache.org/job/PreCommit-SOLR-Build/595/console |
| Powered by | Apache Yetus 0.7.0 http://yetus.apache.org |
This message was automatically generated.
> Backup can fail to read index files w/NoSuchFileException during merges
> (SOLR-11616 regression)
>
>
> Key: SOLR-13872
> URL: https://issues.apache.org/jira/browse/SOLR-13872
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Chris M. Hostetter
>Assignee: Chris M. Hostetter
>Priority: Major
> Attachments: SOLR-13872.patch, SOLR-13872.patch, SOLR-13872.patch,
> SOLR-13872.patch, index_churn.pl
>
>
> SOLR-11616 purports to fix a bug in Solr's backup functionality that causes
> 'NoSuchFileException' errors when attempting to backup an index while it is
> undergoing indexing (and segment merging)
> Although SOLR-11616 is marked with "Fix Version: 7.2" it's pretty easy to
> demonstrate that this bug still exists on master, branch_8x, and even in 7.2
> - so it seems less like the current problem is a "regression" and more that
> the original fix didn't work.
>
> The crux of the problem seems to be concurrency bugs in if/how a commit is
> "reserved" before attempting to copy the files in that commit to the backup
> location.
> A possible work around discussed in more depth in the comments below is to
> update {{solrconfig.xml}} to explicitly configure the {{SolrDeletionPolicy}}
> with either the {{maxCommitsToKeep}} or {{maxCommitAge}} options to ensure
> the commits are kept around long enough for the backup to be created.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9043) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
pavithra kariyawasam created LUCENE-9043: Summary: Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words. Key: LUCENE-9043 URL: https://issues.apache.org/jira/browse/LUCENE-9043 Project: Lucene - Core Issue Type: Improvement Components: modules/analysis Affects Versions: 8.3 Reporter: pavithra kariyawasam Fix For: 5.5.6 Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, SinhalaTokenizer.java, stopwords.txt This component is developed based on three main researches. Sinhala Analyzer, as it word implies it is an enhanced software library to analyze documents which are written in Sinhala language. Sinhala Analyzer has implemented by performing Sinhala morphological analysis. Tokenizing the document content precisely, Removing stopwords accordingly and converting the terms to its base/root form accurately are the main three functionalities of Sinhala Analyzer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9044) Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words.
pavithra kariyawasam created LUCENE-9044: Summary: Currently Lucene doesn't have an analyzer for Sinhala. We have built analyzer which consist of language dependent tokenizer, stemming algorithm and list of stop words. Key: LUCENE-9044 URL: https://issues.apache.org/jira/browse/LUCENE-9044 Project: Lucene - Core Issue Type: Improvement Components: modules/analysis Affects Versions: 8.3 Environment: Lucene Reporter: pavithra kariyawasam Fix For: 5.5.6 Attachments: SinhalaAnalyzer.java, SinhalaStemmer.java, SinhalaTokenizer.java, stopwords.txt This component is developed based on three main researches. Lucene did not have component to analyze Sinhala documents. So our intension is to fill that space with an Analyzer which can analyze Sinhala documents. Sinhala Analyzer has implemented by performing Sinhala morphological analysis. Tokenizing the document content precisely, Removing stopwords accordingly and converting the terms to its base/root form accurately are the main three functionalities of Sinhala Analyzer. These are built by considering the grammatical rules in Sinhala -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13822) Isolated Classloading from packages
[
https://issues.apache.org/jira/browse/SOLR-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973061#comment-16973061
]
ASF subversion and git services commented on SOLR-13822:
Commit e4283153878850b3e48be5da4913079c0492ff9d in lucene-solr's branch
refs/heads/branch_8x from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=e428315 ]
SOLR-13822: Added a testcase and a bug fix
> Isolated Classloading from packages
> ---
>
> Key: SOLR-13822
> URL: https://issues.apache.org/jira/browse/SOLR-13822
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13822.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Design is here:
> [https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?ts=5d86a8ad#]
>
> main features:
> * A new file for packages definition (/packages.json) in ZK
> * Public APIs to edit/read the file
> * The APIs are registered at {{/api/cluster/package}}
> * Classes can be loaded from the package classloader using the
> {{:}} syntax
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13822) Isolated Classloading from packages
[
https://issues.apache.org/jira/browse/SOLR-13822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973058#comment-16973058
]
ASF subversion and git services commented on SOLR-13822:
Commit 550c7296b6f9abc4ca73c16eaf8c6f1d67bbe7d5 in lucene-solr's branch
refs/heads/master from Noble Paul
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=550c729 ]
SOLR-13822: Added a testcase and a bug fix
> Isolated Classloading from packages
> ---
>
> Key: SOLR-13822
> URL: https://issues.apache.org/jira/browse/SOLR-13822
> Project: Solr
> Issue Type: Sub-task
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Ishan Chattopadhyaya
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.4
>
> Attachments: SOLR-13822.patch
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> Design is here:
> [https://docs.google.com/document/d/15b3m3i3NFDKbhkhX_BN0MgvPGZaBj34TKNF2-UNC3U8/edit?ts=5d86a8ad#]
>
> main features:
> * A new file for packages definition (/packages.json) in ZK
> * Public APIs to edit/read the file
> * The APIs are registered at {{/api/cluster/package}}
> * Classes can be loaded from the package classloader using the
> {{:}} syntax
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13841) Provide mappings for jackson annotation @JsonProperty to use Jackson deserializer
[ https://issues.apache.org/jira/browse/SOLR-13841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16973060#comment-16973060 ] ASF subversion and git services commented on SOLR-13841: Commit bb2385025fe8970208330e40e5914f58e6310f5f in lucene-solr's branch refs/heads/branch_8x from Noble Paul [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=bb23850 ] SOLR-13841: Added 'JsonProperty' to forbidden APIs > Provide mappings for jackson annotation @JsonProperty to use Jackson > deserializer > - > > Key: SOLR-13841 > URL: https://issues.apache.org/jira/browse/SOLR-13841 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Noble Paul >Assignee: Noble Paul >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > We can start using annotations in SolrJ to minimize the amount of code we > write & improve readability. Jackson is a widely used library and everyone is > already familiar with it -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
