[jira] [Assigned] (SOLR-13125) Optimize Queries when sorting by router.field
[ https://issues.apache.org/jira/browse/SOLR-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck reassigned SOLR-13125: --- Assignee: Gus Heck > Optimize Queries when sorting by router.field > - > > Key: SOLR-13125 > URL: https://issues.apache.org/jira/browse/SOLR-13125 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Assignee: Gus Heck >Priority: Minor > Attachments: SOLR-13125-no-commit.patch, SOLR-13125.patch, > SOLR-13125.patch, SOLR-13125.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries sorted by router.field(the field which TRA > uses to route the data to the correct collection) has emerged. > Perhaps, in queries which are sorted by router.field, Solr could be smart > enough to wait for the more recent collections, and in case the limit was > reached cancel other queries(or just not block and wait for the results)? > For example: > When querying a TRA which with a filter on a different field than > router.field, but sorting by router.field desc, limit=100. > Since this is a TRA, solr will issue queries for all the collections in the > alias. > But to optimize this particular type of query, Solr could wait for the most > recent collection in the TRA, see whether the result set matches or exceeds > the limit. If so, the query could be returned to the user without waiting for > the rest of the shards. If not, the issuing node will block until the second > query returns, and so forth, until the limit of the request is reached. > This might also be useful for deep paging, querying each collection and only > skipping to the next once there are no more results in the specified > collection. > Thoughts or inputs are always welcome. > This is just my two cents, and I'm always happy to brainstorm. > Thanks in advance. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11492) More Modern cloud dev script
[ https://issues.apache.org/jira/browse/SOLR-11492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16926689#comment-16926689 ] Gus Heck commented on SOLR-11492: - current versions living here until I get some feedback and verification that works on machines other than mine :) [https://gist.github.com/nsoft/d05d9e3f019ac823c0fe418f1f4bee5d] > More Modern cloud dev script > > > Key: SOLR-11492 > URL: https://issues.apache.org/jira/browse/SOLR-11492 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Minor > Attachments: cloud.sh, cloud.sh, cloud.sh, cloud.sh, cloud.sh, > cloud.sh > > > Most of the scripts in solr/cloud-dev do things like start using java -jar > and other similarly ancient techniques. I recently decided I really didn't > like that it was a pain to setup a cloud to test a patch/feature and that > often one winds up needing to blow away existing testing so working on more > than one thing at a time is irritating... so here's a script I wrote, if > folks like it I'd be happy for it to be included in solr/cloud-dev -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13655) Cut Over Collections.unmodifiableSet usages to Set.*
[ https://issues.apache.org/jira/browse/SOLR-13655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16922971#comment-16922971 ] Gus Heck commented on SOLR-13655: - I could have sworn there was a discussion that lead to the conclusion that Java 9+ idioms should wait until the first 9.0 to avoid complicating merges but I cant' for the life of me find it by searching in gmail... > Cut Over Collections.unmodifiableSet usages to Set.* > > > Key: SOLR-13655 > URL: https://issues.apache.org/jira/browse/SOLR-13655 > Project: Solr > Issue Type: Improvement >Reporter: Atri Sharma >Priority: Major > Fix For: master (9.0) > > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13696) DimensionalRoutedAliasUpdateProcessorTest / RoutedAliasUpdateProcessorTest failures due commitWithin/openSearcher delays
[
https://issues.apache.org/jira/browse/SOLR-13696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16908689#comment-16908689
]
Gus Heck commented on SOLR-13696:
-
Thx for the write up and suggestions. I will look as soon as I can, but this
weekend is not looking likely due to weekend commitments & visiting family.
Possibly by Monday or Tuesday I'll just declare a solr day (or half day) for
myself, there are several things I want to look at that keep getting pushed.
Routed Alias tests in general do seem to run up against the timing sorts of
issues frequently. I've had no end of trouble with various bits of code that
need to wait for a collection to be available, or a zookeeper related change to
be visible (or both) etc. The existing wait methods for that stuff all seem to
fail some small fraction of a percent of the time... for example SOLR-13059...
which is among the things on my list to go back and check the status of and see
if it's improved or if anything I've learned since makes the solution more
obvious...
> DimensionalRoutedAliasUpdateProcessorTest / RoutedAliasUpdateProcessorTest
> failures due commitWithin/openSearcher delays
>
>
> Key: SOLR-13696
> URL: https://issues.apache.org/jira/browse/SOLR-13696
> Project: Solr
> Issue Type: Test
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Hoss Man
>Assignee: Gus Heck
>Priority: Major
> Attachments: thetaphi_Lucene-Solr-8.x-MacOSX_272.log.txt
>
>
> Recent jenkins failure...
> Build: https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-MacOSX/272/
> Java: 64bit/jdk1.8.0 -XX:-UseCompressedOops -XX:+UseParallelGC
> {noformat}
> Stack Trace:
> java.lang.AssertionError: expected:<16> but was:<15>
> at
> __randomizedtesting.SeedInfo.seed([DB6DC28D5560B1D2:E295833E1541FDB9]:0)
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.solr.update.processor.DimensionalRoutedAliasUpdateProcessorTest.assertCatTimeInvariants(DimensionalRoutedAliasUpdateProcessorTest.java:677
> )
> at
> org.apache.solr.update.processor.DimensionalRoutedAliasUpdateProcessorTest.testTimeCat(DimensionalRoutedAliasUpdateProcessorTest.java:282)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {noformat}
> Digging into the logs, the problem appears to be in the way the test
> verifies/assumes docs have been committed.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Assigned] (SOLR-13059) TimeRoutedAliasUpdateProcessorTest rarely fails to see collection just created
[
https://issues.apache.org/jira/browse/SOLR-13059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck reassigned SOLR-13059:
---
Assignee: Gus Heck
> TimeRoutedAliasUpdateProcessorTest rarely fails to see collection just created
> --
>
> Key: SOLR-13059
> URL: https://issues.apache.org/jira/browse/SOLR-13059
> Project: Solr
> Issue Type: Bug
> Components: Tests
>Affects Versions: 8.0
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
>
> This issue is for tracking down and fixing this stack trace observed during
> SOLR-13051:
> {code:java}
> [junit4] ERROR 11.2s | TimeRoutedAliasUpdateProcessorTest.testSliceRouting <<<
> [junit4] > Throwable #1:
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException:
> Error from server at http://127.0.0.1:32891/solr: no core retrieved for
> testSliceRouting
> [junit4] > at
> __randomizedtesting.SeedInfo.seed([D9B32625DD304967:E823AAC97966B64E]:0)
> [junit4] > at
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteExecutionException.create(HttpSolrClient.java:829)
> [junit4] > at
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:620)
> [junit4] > at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255)
> [junit4] > at
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244)
> [junit4] > at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:484)
> [junit4] > at
> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:414)
> [junit4] > at
> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1110){code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13242) RegexReplaceProcessorFactory not making accurate replacement
[
https://issues.apache.org/jira/browse/SOLR-13242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16905193#comment-16905193
]
Gus Heck commented on SOLR-13242:
-
RegexReplaceUpdateProcessorFactory is a very simple class. As noted above it's
just a wrapper around Matcher.replaceAll()
{code:java}
return valueMutator(getSelector(), next, src -> {
if (src instanceof CharSequence) {
CharSequence txt = (CharSequence) src;
return pattern.matcher(txt).replaceAll(replacement);
}
return src;
});
{code}
I notice this issue is dealing with patterns involving newlines and whitespace.
Whitespace can be a hairy and complicated thing, since it's not rendered
visibly unless your editor either fails to recognize it and prints ?'s... so
the better, more complete your editor, the less likely you are to see the
problem :). You often think you know what you're looking at but don't. It seems
possible that you are running into unusual white space chars that break your
pattern. One can see how fiddly this gets from things seen in Pattern.java:
||Predefined character classes||
|{{.}}|Any character (may or may not match [line
terminators|https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html#lt])|
|{{\d}}|A digit: {{[0-9]}}|
|{{\D}}|A non-digit: {{[^0-9]}}|
|{{\h}}|A horizontal whitespace character: {{[
\t\xA0\u1680\u180e\u2000-\u200a\u202f\u205f\u3000]}}|
|{{\H}}|A non-horizontal whitespace character: {{[^\h]}}|
|{{\s}}|A whitespace character: {{[ \t\n\x0B\f\r]}}|
|{{\S}}|A non-whitespace character: {{[^\s]}}|
|{{\v}}|A vertical whitespace character: {{[\n\x0B\f\r\x85\u2028\u2029]}}|
|{{\V}}|A non-vertical whitespace character: {{[^\v]}}|
|{{\w}}|A word character: {{[a-zA-Z_0-9]}}|
|{{\W}}|A non-word character: {{[^\w]}}|
(\h|\v) appears to catch a lot more types of whitespace than \s for example.
See also (with (?U)which turns on the unicode character class feature).
|{{\p\{Blank}}}|A space or a tab:
{{[\p\{IsWhite_Space}&&[^\p\{gc=Zl}\p\{gc=Zp}\x0a\x0b\x0c\x0d\x85]]}}|
The comments above indicating that \W and \h instead of \s work better for the
OP seem to support the hypothesis that the issue was caused by was oddball
white space in the data, not a bug in Solr. [~edwinyeozl] can we resolve "this
as not a problem"?
> RegexReplaceProcessorFactory not making accurate replacement
>
>
> Key: SOLR-13242
> URL: https://issues.apache.org/jira/browse/SOLR-13242
> Project: Solr
> Issue Type: Bug
>Affects Versions: 7.6, 7.7, 7.7.1
>Reporter: Edwin Yeo Zheng Lin
>Priority: Major
> Labels: regex, solr
>
> We are using the RegexReplaceProcessorFactory, and have tried with all of the
> following configurations in solrconfig.xml:
>
>
> content
> (\s*\r?\n)\{2,}
>
> true
>
>
> content
> ([ \s]*\r?\n)\{2,}
>
> true
>
>
> content
> (\s*\n)\{2,}
>
> true
>
>
> content
> (\n\s*)\{2,}
>
> true
>
>
> The regex pattern of (\s*\r?\n)\{2,}, ([ \s]*\r?\n)\{2,}, (\s*\n)\{2,} and
> (\n\s*)\{2,} are working perfectly in [regex101.com|http://regex101.com/], in
> which all the \n will be replaced by only two
> However, in Solr, there are cases (in Example 2 and 3 below) that has four
> in a row. This should not be the case, as we have already set it to
> replace by two regardless of how many \n are there in a row.
>
>
> *Example 1: The sentence that the above regex pattern is working correctly*
> *Original content in EML [file:*|file://%2A/]
> Dear Sir,
>
> I am terminating
> *Original content:* Dear Sir, \n\n \n \n\n I am terminating
> *Index content:* Dear Sir, I am terminating
>
> *Example 2: The sentence that the above regex pattern is partially working
> (as you can see, instead of 2 , there are 4 )*
> *Original content in EML [file:*|file://%2A/]
> _exalted_
> _Psalm 89:17_
>
> 3 Choa Chu Kang Avenue 4
> *Original content:* exalted \n \n\n Psalm 89:17 \n\n \n\n 3 Choa Chu
> Kang Avenue 4, Singapore
> *Index content:* exalted Psalm 89:17 3 Choa Chu
> Kang Avenue 4, Singapore
>
> *Example 3: The sentence that the above regex pattern is partially working
> (as you can see, instead of 2 , there are 4 )*
> *Original content in EML [file:*|file://%2A/]
> [http://www.concordpri.moe.edu.sg/]
>
>
>
>
> On Tue, Dec 18, 2018 at 10:07 AM
> *Original content:* [http://www.concordpri.moe.edu.sg/] \n\n \n\n \n \n\n
> \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n \n\n\n \n\n\n On Tue, Dec 18, 2018
> at 10:07 AM
> *Index content:* [http://www.concordpri.moe.edu.sg/] On
> Tue, Dec 18, 2018 at 10:07 AM
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (SOLR-12801) Fix the tests, remove BadApples and AwaitsFix annotations, improve env for test development.
[ https://issues.apache.org/jira/browse/SOLR-12801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16903021#comment-16903021 ] Gus Heck commented on SOLR-12801: - [~aninditagupta] this issue is about fixing the unit tests. SOLR-13457 is probably a better place to discuss issues with hard coded timeouts. > Fix the tests, remove BadApples and AwaitsFix annotations, improve env for > test development. > > > Key: SOLR-12801 > URL: https://issues.apache.org/jira/browse/SOLR-12801 > Project: Solr > Issue Type: Task >Reporter: Mark Miller >Assignee: Mark Miller >Priority: Critical > Time Spent: 1h > Remaining Estimate: 0h > > A single issue to counteract the single issue adding tons of annotations, the > continued addition of new flakey tests, and the continued addition of > flakiness to existing tests. > Lots more to come. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
[
https://issues.apache.org/jira/browse/SOLR-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900348#comment-16900348
]
Gus Heck commented on SOLR-13559:
-
The last failure report has one fail, but it's the 4th week in a row so this is
getting bad appled now. The more recent failure mode seems to be
{code:java}
[junit4] FAILURE 4.74s J1 | AliasIntegrationTest.testClusterStateProviderAPI
<<<
[junit4]> Throwable #1: java.lang.AssertionError: should be a routed
alias
[junit4]>at
__randomizedtesting.SeedInfo.seed([D1FEFD814D87287B:CE2961AD3E8CD130]:0)
[junit4]>at
org.apache.solr.cloud.AliasIntegrationTest.testClusterStateProviderAPI(AliasIntegrationTest.java:315)
[junit4]>at java.lang.Thread.run(Thread.java:748)
{code}
which appears to stem from a property being set, but not observed, probably
meaning that waitForAliasesUpdate() isn't really doing it's job.
Current report on fucit.org only shows one fail on 8.x so It's somewhat unclear
if this is 8x only or generally a problem. -Dtests.seed=D1FEFD814D87287B from
the recorded failure does not reproduce the issue
> AliasIntegrationTest.testClusterStateProviderAPI fails to often
> ---
>
> Key: SOLR-13559
> URL: https://issues.apache.org/jira/browse/SOLR-13559
> Project: Solr
> Issue Type: Bug
> Components: Tests
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Priority: Minor
>
> Recent failure rates for AliasIntegrationTest.testClusterStateProviderAPI
> have been around 4% which is too high.
> (http://fucit.org/solr-jenkins-reports/failure-report.html). I've beasted 100
> runs a couple times and not reproduced it but then hit a failure in it during
> a normal test run today, so I'm going to start this ticket and record the
> trace. I have yet to dig through the zips for the logs from the builds.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889170#comment-16889170
]
Gus Heck commented on SOLR-11266:
-
I do agree that it's not a bug by the way :)
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Improvement
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-11266.patch
>
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-11266:
Issue Type: Improvement (was: Bug)
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Improvement
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-11266.patch
>
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16889152#comment-16889152
]
Gus Heck commented on SOLR-11266:
-
{quote}More than someone trying to write a context-free Solr client I'd say ;)
{quote}
You mean like jquery? with it's default intelligent guess setting?
{panel:title= dataType (default: Intelligent Guess (xml, json, script, or
html))}
Type: [String|http://api.jquery.com/Types/#String]
The type of data that you're expecting back from the server. If none is
specified, jQuery will try to infer it based on the MIME type of the response
(an XML MIME type will yield XML, in 1.4 JSON will yield a JavaScript object,
in 1.4 script will execute the script, and anything else will be returned as a
string).
{panel}
Seen here in dev console on current solr home page with a collection named foo:
{code:java}
$.ajax("/solr/foo/query?q=*:*").done(function(data) {console.log(typeof data)});
string
{code}
Sure you can specify the dataType as json (i.e. hardcode the expected type, the
most common solution)... but by default you get a string, and so first contact
with solr for most jquery ui programmers is to hit an error and discover that
the jquery defaults don't work...
{quote}(but it's not incorrect to have json formatted text in a plain text HTTP
response, and I disagree that this issue should be categorized as a bug.)
Although one can argue that application/json is *more* appropriate given that
it's more specific.
{quote}
absolutely true from the HTTP spec perspective, anything that can handle
text/plain will not break when receiving application/json, but not true from a
JSON perspective where RFC 4627 and 7159 do state that the correct media type
is application/json. So one can say correctly that we serve valid text/plain.
One can't say that we are serving valid JSON, because we haven't served it with
the correct media type.
It is true that the error we are making is so common (see also: text/json
text/x-json, etc) that there are easy workarounds built into many libraries.
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Bug
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
> Attachments: SOLR-11266.patch
>
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888964#comment-16888964
]
Gus Heck commented on SOLR-11266:
-
{quote}Adding different config for browser would be maintenance headache.
{quote}
Yeah, if we don't want to pay that cost I'm fine with it. Just a suggestion how
we could keep both. But yeah it's not free.
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Bug
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) 2 Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1692#comment-1692 ] Gus Heck commented on SOLR-13375: - Noticed I seem to have forgotten to do a CHANGES entry too, I'll fix that now > 2 Dimensional Routed Aliases > > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) 2 Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888401#comment-16888401 ] Gus Heck commented on SOLR-13375: - Thanks, I'll look into it > 2 Dimensional Routed Aliases > > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887966#comment-16887966
]
Gus Heck commented on SOLR-11266:
-
For reference, this is the override from the configs:
{code:xml}
text/plain; charset=UTF-8
{code}
We could distribute _default with this corrected (removed), and have a
_browser_example config also installed and available by default that does have
this override. A name like that should make it pretty clear which to select
when going through examples in the ref guide or other publications, but not
become what people automatically use as a starting point.
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Bug
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-11266) V2 API returning wrong content-type
[
https://issues.apache.org/jira/browse/SOLR-11266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886363#comment-16886363
]
Gus Heck commented on SOLR-11266:
-
This behavior has been a peeve of mine... I've always found it irritating that
the content type is not appropriate for the content. This forces clients to be
written to make hard coded assumptions about content type (the most common
solution), do checks to find out what was requested on the request line (or
included in the POST data from the form), or to maintain state across the
request or do any one of a number of other non-standard things rather than
simply choose a parser based on content type. And that's if one has good
control over that communications code. Could be even more fun if the client
wants to use 3rd party code that refuses to ignore the header. I'm not of the
opinion that users looking at a response in a browser are our main target
audience. Javascript, Java, PHP and Python code probably constitute 1000x more
use cases than humans in browser window inspection, which is mostly an initial
development thing. If plain text is desired, that should be produced as the
result of an override. Looking back at that the referenced issue, the number
of folks rooting for application/json seemed way larger than those insisting on
text/plain
Standards are standards so that people can rely on them, and less inane
glue-code is needed for systems to talk to each other. I never like to see
software (be it open source or closed) half supporting or mutating or
"extending" standards. I'm very +1 that we target returning sane content types
that match the content actually returned for 9.x and provide a text/plain back
compatibility option for the request params and also as a global config
somewhere. It should not take special configuration to turn on "standards mode".
We should also be able to do Accept: negotiation and not require a url param
relating to content type at all, at which point a browser that didn't accept
json would likely fall back to text/plain...
> V2 API returning wrong content-type
> ---
>
> Key: SOLR-11266
> URL: https://issues.apache.org/jira/browse/SOLR-11266
> Project: Solr
> Issue Type: Bug
> Components: v2 API
>Reporter: Ishan Chattopadhyaya
>Priority: Major
>
> The content-type of the returned value is wrong in many places. It should
> return "application/json", but instead returns "application/text-plan".
> Here's an example:
> {code}
> [ishan@t430 ~] $ curl -v
> "http://localhost:8983/api/collections/products/select?q=*:*=0;
> * Trying 127.0.0.1...
> * TCP_NODELAY set
> * Connected to localhost (127.0.0.1) port 8983 (#0)
> > GET /api/collections/products/select?q=*:*=0 HTTP/1.1
> > Host: localhost:8983
> > User-Agent: curl/7.51.0
> > Accept: */*
> >
> < HTTP/1.1 200 OK
> < Content-Type: text/plain;charset=utf-8
> < Content-Length: 184
> <
> {
> "responseHeader":{
> "zkConnected":true,
> "status":0,
> "QTime":1,
> "params":{
> "q":"*:*",
> "rows":"0"}},
> "response":{"numFound":260,"start":0,"docs":[]
> }}
> * Curl_http_done: called premature == 0
> * Connection #0 to host localhost left intact
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13125) Optimize Queries when sorting by router.field
[ https://issues.apache.org/jira/browse/SOLR-13125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886151#comment-16886151 ] Gus Heck commented on SOLR-13125: - The idea behind this patch is interesting. Unless I misunderstand the intent, the idea is to short circuit the response collection when the TRA collection names tell us that further responses will all return docs that are too far down the result list to ever be included. Unfortunately I don't think this patch does that. Issues I see: * This patch overrides finishStage() instead of handleResponses, which means that by the time your logic runs all responses have been received. * I don't see logic to handle values for the start parameter * Also not sure I like the tests checking debug messages rather than actual code behavior. That could get out of sync. In any case, it's unclear to me if this can be handled in a search component without core changes, even if you override handleResponses() instead, you can't stop SearchHandler from looping and attempting to take() the results of every request that was sent (unless you throw an exception, but that wont be good). What you would need to do is somehow influence the futures that solr is waiting on to return early and empty once your request has been filled up from the most recent collections. (see org/apache/solr/handler/component/HttpShardHandler.java:281). Baring that, you could perhaps find a way to empty the pending queue, but that means you still have to wait for at least one uninteresting request to complete. The futures themselves would be waiting on the org/apache/solr/handler/component/HttpShardHandler.java:201. call to makeLoadBalancedRequest(), so I think this optimization requires the addition of an explicit short-circuit enabling hook. Possibly this could be a new method for SearchComponents to override, but we need to think some about how that would play with assumptions of existing code some. > Optimize Queries when sorting by router.field > - > > Key: SOLR-13125 > URL: https://issues.apache.org/jira/browse/SOLR-13125 > Project: Solr > Issue Type: Sub-task >Reporter: mosh >Priority: Minor > Attachments: SOLR-13125-no-commit.patch, SOLR-13125.patch, > SOLR-13125.patch, SOLR-13125.patch > > Time Spent: 10m > Remaining Estimate: 0h > > We are currently testing TRA using Solr 7.7, having >300 shards in the alias, > with much growth in the coming months. > The "hot" data(in our case, more recent) will be stored on stronger > nodes(SSD, more RAM, etc). > A proposal of optimizing queries sorted by router.field(the field which TRA > uses to route the data to the correct collection) has emerged. > Perhaps, in queries which are sorted by router.field, Solr could be smart > enough to wait for the more recent collections, and in case the limit was > reached cancel other queries(or just not block and wait for the results)? > For example: > When querying a TRA which with a filter on a different field than > router.field, but sorting by router.field desc, limit=100. > Since this is a TRA, solr will issue queries for all the collections in the > alias. > But to optimize this particular type of query, Solr could wait for the most > recent collection in the TRA, see whether the result set matches or exceeds > the limit. If so, the query could be returned to the user without waiting for > the rest of the shards. If not, the issuing node will block until the second > query returns, and so forth, until the limit of the request is reached. > This might also be useful for deep paging, querying each collection and only > skipping to the next once there are no more results in the specified > collection. > Thoughts or inputs are always welcome. > This is just my two cents, and I'm always happy to brainstorm. > Thanks in advance. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) 2 Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884509#comment-16884509 ] Gus Heck commented on SOLR-13375: - Patch with docs & cleanup ready for commit in a day or two unless I hear objections. Support only for 2D Dimensional Routed Aliases since there are folks waiting with the 2D use case in mind and I need to attend to other work before I sort out the 3+ dimension cases. This patch flat out forbids more than 2 dimensions so nobody tries to use what's not yet working. The V1 and V2 API's already fully support the creation of N dimensions, but the collection creation isn't quite behaving itself in the 3 category case I tested while writing docs so the collections API portion of this is not going to change other than eventually it won't error out for more than 2 dims. > 2 Dimensional Routed Aliases > > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13375) 2 Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Attachment: SOLR-13375.patch > 2 Dimensional Routed Aliases > > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (LUCENE-8913) Reproducing failure in various TestLatLon* equals/hashcode tests
Gus Heck created LUCENE-8913:
Summary: Reproducing failure in various TestLatLon*
equals/hashcode tests
Key: LUCENE-8913
URL: https://issues.apache.org/jira/browse/LUCENE-8913
Project: Lucene - Core
Issue Type: Bug
Components: core/other
Affects Versions: master (9.0)
Reporter: Gus Heck
Bumped into this while running tests locally
ant clean test -Dtests.seed=41D0C5A80C823307 -Dtests.slow=true
-Dtests.badapples=true -Dtests.locale=es-CL -Dtests.timezone=Pacific/Rarotonga
-Dtests.asserts=true -Dtests.file.encoding=UTF-8
reliably produces:
{code:java}
Tests with failures [seed: 41D0C5A80C823307]:
[junit4] -
org.apache.lucene.document.TestLatLonPointShapeQueries.testBoxQueryEqualsAndHashcode
[junit4] -
org.apache.lucene.document.TestLatLonMultiPointShapeQueries.testBoxQueryEqualsAndHashcode
[junit4] -
org.apache.lucene.document.TestLatLonLineShapeQueries.testBoxQueryEqualsAndHashcode
[junit4] -
org.apache.lucene.document.TestLatLonPolygonShapeQueries.testBoxQueryEqualsAndHashcode
[junit4] -
org.apache.lucene.document.TestLatLonMultiPolygonShapeQueries.testBoxQueryEqualsAndHashcode
[junit4] -
org.apache.lucene.document.TestLatLonMultiLineShapeQueries.testBoxQueryEqualsAndHashcode{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13628) N - Dimensional Routed Aliases
Gus Heck created SOLR-13628: --- Summary: N - Dimensional Routed Aliases Key: SOLR-13628 URL: https://issues.apache.org/jira/browse/SOLR-13628 Project: Solr Issue Type: New Feature Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Reporter: Gus Heck Assignee: Gus Heck This is a follow on from SOLR-13375, which is working for 2 dimensional cases but has some bugs remaining for 3 dimensions and up. The 2 D case is expected to be the core case and the most useful case so I'm splitting this out into it's own ticket so I can get the current code out where it can be useful, while this remaining bit is sorted out. 13375 documentation will reflect the fact that only 2 dimensions are presently supported. I do not expect any significant changes in the V1/V2 API for DRA's to enable this support. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13375) 2 Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Summary: 2 Dimensional Routed Aliases (was: Dimensional Routed Aliases) > 2 Dimensional Routed Aliases > > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883163#comment-16883163 ] Gus Heck commented on SOLR-13375: - Patch with tests and fixes to make the tests pass. Things yet to do: * Documentation * Review the javadocs and comments in code many of which may have become obsolete After that I'll look to commit this to master and 8x for inclusion in 8.3 > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Attachment: SOLR-13375.patch > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch, > SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878979#comment-16878979 ] Gus Heck commented on SOLR-13375: - First potentially functional patch. Lots of refactoring to move logic into the routed alias classes from the maintain Cmd classes, which are now consolidated to a single MaintainRoutedAlias class. Also refactored many longer methods into smaller chunks. The basic strategy here is to use a specialized subclass of the primary routed alias classes to provide the context for answering the question of whether or not collections need to be created. I also altered the basic logic such that the routed aliases fully calculate the collection to which the document should be routed and then push that "target" collection to the maintain command repeatedly until the collection required has been created. This simplifies the situation which previously had MaintainCategoryRoutedAliasCmd working off of the value encountered in the document and the MaintainTimeRoutedAlias was just marching the collections forward without knowledge of the end-state. The imbalance in those strategies needed to be resolved to keep DRA's tractable. Another notable abstraction added is a notion of "actions" that are requested by each routed alias during the execution of the MaintainRoutedAliasCmd. In the case of DRA's these are generated by each sub-dimension and collated into a final set of actions. by the DRA. Additionally I hit a very time consuming bug with tests, where I eventually realized that the problem is that the results of an admin command become visible (to the test) before the execution of the command is entirely completed, and the test that has waited for a collection to be visible can begin to shut down while an Async operation is still in progress. This can lead to never being released from the watcher.await(timeout); call in OverseerTaskQueue.offer never releasing (and then the shutdown cycle that is waiting for the core async thread to terminate waits until the timeout expires). This only showed up if I saturated my CPU and then only ablut 20% of the time. The sneaky thing about this is if you beasted it and went to bed or went to lunch it would complete successfully because of the timeout, but the time it took to do so was ridiculous if you were waiting for it. A 5 second Thread.sleep() as the last line of the test reliably resolved this, but not being happy with that I added a count of pending overseerTasks and a allowOverseerPendingTasksToComplete() method to OverseerTaskQueue and the first thing that happens on CoreContainer.shutdown is it calls the new method (which of course first prohibits new tasks from being queued... though I'm not sure if the exception thrown to threads that try is ideal...). Once the in-progress tasks finish shutdown proceeds normally. This completely solved the problems with my async collection creation tests. [^SOLR-13375.patch] > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands,
[jira] [Updated] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Attachment: SOLR-13375.patch > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
[ https://issues.apache.org/jira/browse/SOLR-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878935#comment-16878935 ] Gus Heck commented on SOLR-13559: - Interestingly the failure rate went down to 2% and is only failing in 8x now hmm. > AliasIntegrationTest.testClusterStateProviderAPI fails to often > --- > > Key: SOLR-13559 > URL: https://issues.apache.org/jira/browse/SOLR-13559 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Gus Heck >Priority: Minor > > Recent failure rates for AliasIntegrationTest.testClusterStateProviderAPI > have been around 4% which is too high. > (http://fucit.org/solr-jenkins-reports/failure-report.html). I've beasted 100 > runs a couple times and not reproduced it but then hit a failure in it during > a normal test run today, so I'm going to start this ticket and record the > trace. I have yet to dig through the zips for the logs from the builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13583) Impossible to delete a collection with the same name as an existing alias
[
https://issues.apache.org/jira/browse/SOLR-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878725#comment-16878725
]
Gus Heck edited comment on SOLR-13583 at 7/4/19 2:55 PM:
-
Tested just for fun... and DELETE
localhost:8983/api/c/dra_test2?followAliases=true gave me a 500 not a 400 (and
yes the first collection name in the DRA there is a bug I'm presently squashing
:) ):
{code:java}
{
"responseHeader": {
"status": 500,
"QTime": 38
},
"Operation delete caused exception:":
"java.lang.IllegalArgumentException:java.lang.IllegalArgumentException: Simple
alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"exception": {
"msg": "Simple alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"rspCode": -1
},
"error": {
"metadata": [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"org.apache.solr.common.SolrException"
],
"msg": "Simple alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"trace": "org.apache.solr.common.SolrException: Simple alias
'dra_test2' points to more than 1 collection: [dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]\n\tat
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat
org.apache.solr.handler.admin.CollectionHandlerApi$1.invoke(CollectionHandlerApi.java:61)\n\tat
org.apache.solr.handler.admin.BaseHandlerApiSupport$1.call(BaseHandlerApiSupport.java:116)\n\tat
org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:341)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:769)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
[jira] [Commented] (SOLR-13583) Impossible to delete a collection with the same name as an existing alias
[
https://issues.apache.org/jira/browse/SOLR-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878725#comment-16878725
]
Gus Heck commented on SOLR-13583:
-
Tested just for fun... and DELETE
localhost:8983/api/c/dra_test2?followAliases=true gave me a 500 not a 400 (and
yes the first collection name in the DRA there is a bug I'm presently squashing
:) ):
{code:java}
{
"responseHeader": {
"status": 500,
"QTime": 38
},
"Operation delete caused exception:":
"java.lang.IllegalArgumentException:java.lang.IllegalArgumentException: Simple
alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"exception": {
"msg": "Simple alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"rspCode": -1
},
"error": {
"metadata": [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"org.apache.solr.common.SolrException"
],
"msg": "Simple alias 'dra_test2' points to more than 1 collection:
[dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]",
"trace": "org.apache.solr.common.SolrException: Simple alias
'dra_test2' points to more than 1 collection: [dra_test2dra_test2__CRA__catA,
dra_test2__CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP__TRA__2019-01-01]\n\tat
org.apache.solr.client.solrj.SolrResponse.getException(SolrResponse.java:53)\n\tat
org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:280)\n\tat
org.apache.solr.handler.admin.CollectionHandlerApi$1.invoke(CollectionHandlerApi.java:61)\n\tat
org.apache.solr.handler.admin.BaseHandlerApiSupport$1.call(BaseHandlerApiSupport.java:116)\n\tat
org.apache.solr.api.V2HttpCall.handleAdmin(V2HttpCall.java:341)\n\tat
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:769)\n\tat
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:529)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:423)\n\tat
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:350)\n\tat
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)\n\tat
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)\n\tat
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)\n\tat
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)\n\tat
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)\n\tat
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)\n\tat
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)\n\tat
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:152)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)\n\tat
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)\n\tat
org.eclipse.jetty.server.Server.handle(Server.java:505)\n\tat
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)\n\tat
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)\n\tat
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)\n\tat
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)\n\tat
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)\n\tat
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875345#comment-16875345 ] Gus Heck commented on SOLR-13375: - Another WIP patch, now solving the v2 api issue via an implementation of toMap() on the SolrParams anon wrapper Both work, and all tests pass (but the DRA they create still isn't functional, that's next). > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Attachment: SOLR-13375.patch > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch, SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13583) Impossible to delete a collection with the same name as an existing alias
[
https://issues.apache.org/jira/browse/SOLR-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875002#comment-16875002
]
Gus Heck edited comment on SOLR-13583 at 6/28/19 3:35 PM:
--
Also this issue should probably clarify the intended behavior relative to
[~erickerickson]'s concerns expressed in: SOLR-13122
My $0.02: The current situation of an alias pointing to an alias but only one
level deep is strange. In 9.0 We should either support nesting (arbitrary
depth) or not support it at all. Certainly nesting imposes significant risks
and more work, and I think if we do follow aliases more than one level, we
should provide an additional attribute followAliasMaxDepth (default=1), and
then fail with if it finds some depth greater than the specified maximum,
returning a list of potentially affected collections, aliases involved, and the
depth required to run the command. This is a lot of work, and so not following
aliases in certain documented cases (or complete rollback) to get this ticket
resolved quickly is probably a good option.
was (Author: gus_heck):
Also this issue should probably clarify the intended behavior relative to
[~erickerickson]'s concerns expressed in: SOLR-13122
My $0.02: The current situation of an alias pointing to an alias but only one
level deep is strange. In 9.0 We should either support nesting (arbitrary
depth) or not support it at all. Certainly nesting imposes significant risks
and more work, and I think if we do follow aliases more than one level, we
should provide require an additional attribute followAliasMaxDepth
(default=1), and then fail with if it finds some depth greater than the
specified maximum, returning a list of potentially affected collections,
aliases involved, and the depth required to run the command. This is a lot of
work, and so not following aliases in certain documented cases (or complete
rollback) to get this ticket resolved quickly is probably a good option.
> Impossible to delete a collection with the same name as an existing alias
> -
>
> Key: SOLR-13583
> URL: https://issues.apache.org/jira/browse/SOLR-13583
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.1, 8.1.1
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Blocker
> Fix For: 8.1.2
>
>
> SOLR-13262 changed the behavior of most collection admin commands so that
> they always resolve aliases by default. In most cases this is desireable
> behavior but it also prevents executing commands on the collections that have
> the same name as an existing alias (which usually points to a different
> collection).
> This behavior also breaks the REINDEXCOLLECTION command with
> {{removeSource=true,}} which can also lead to data loss.
> This issue can be resolved by adding either an opt-in or opt-out flag to the
> collection admin commands that specifies whether the command should attempt
> resolving the provided name as an alias first. From the point of view of ease
> of use this could be an opt-out option, from the point of view of data safety
> this could be an opt-in option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13583) Impossible to delete a collection with the same name as an existing alias
[
https://issues.apache.org/jira/browse/SOLR-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875002#comment-16875002
]
Gus Heck commented on SOLR-13583:
-
Also this issue should probably clarify the intended behavior relative to
[~erickerickson]'s concerns expressed in: SOLR-13122
My $0.02: The current situation of an alias pointing to an alias but only one
level deep is strange. In 9.0 We should either support nesting (arbitrary
depth) or not support it at all. Certainly nesting imposes significant risks
and more work, and I think if we do follow aliases more than one level, we
should provide require an additional attribute followAliasMaxDepth
(default=1), and then fail with if it finds some depth greater than the
specified maximum, returning a list of potentially affected collections,
aliases involved, and the depth required to run the command. This is a lot of
work, and so not following aliases in certain documented cases (or complete
rollback) to get this ticket resolved quickly is probably a good option.
> Impossible to delete a collection with the same name as an existing alias
> -
>
> Key: SOLR-13583
> URL: https://issues.apache.org/jira/browse/SOLR-13583
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.1, 8.1.1
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Blocker
> Fix For: 8.1.2
>
>
> SOLR-13262 changed the behavior of most collection admin commands so that
> they always resolve aliases by default. In most cases this is desireable
> behavior but it also prevents executing commands on the collections that have
> the same name as an existing alias (which usually points to a different
> collection).
> This behavior also breaks the REINDEXCOLLECTION command with
> {{removeSource=true,}} which can also lead to data loss.
> This issue can be resolved by adding either an opt-in or opt-out flag to the
> collection admin commands that specifies whether the command should attempt
> resolving the provided name as an alias first. From the point of view of ease
> of use this could be an opt-out option, from the point of view of data safety
> this could be an opt-in option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13583) Impossible to delete a collection with the same name as an existing alias
[
https://issues.apache.org/jira/browse/SOLR-13583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874966#comment-16874966
]
Gus Heck commented on SOLR-13583:
-
SOLR-13262 has been released, so this needs to be fixed in 8x. We can't change
the basic way aliases work (by requiring uniqueness) until a major version
(9.0) IMHO since that's not backwards compatible. I think we probably need to
bolt on a followAliases=true to enable what was released on 13262 or roll 13262
back. Then we can change things fundamentally in 9.0 if we desire (in another
ticket).
> Impossible to delete a collection with the same name as an existing alias
> -
>
> Key: SOLR-13583
> URL: https://issues.apache.org/jira/browse/SOLR-13583
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Affects Versions: 8.1, 8.1.1
>Reporter: Andrzej Bialecki
>Assignee: Andrzej Bialecki
>Priority: Blocker
> Fix For: 8.1.2
>
>
> SOLR-13262 changed the behavior of most collection admin commands so that
> they always resolve aliases by default. In most cases this is desireable
> behavior but it also prevents executing commands on the collections that have
> the same name as an existing alias (which usually points to a different
> collection).
> This behavior also breaks the REINDEXCOLLECTION command with
> {{removeSource=true,}} which can also lead to data loss.
> This issue can be resolved by adding either an opt-in or opt-out flag to the
> collection admin commands that specifies whether the command should attempt
> resolving the provided name as an alias first. From the point of view of ease
> of use this could be an opt-out option, from the point of view of data safety
> this could be an opt-in option.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[
https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874281#comment-16874281
]
Gus Heck commented on SOLR-13122:
-
wandering Off topic a bit but
{quote} Say somebody deletes an alias and we follow it and delete the
underlying collection.
{quote}
I think this *should* be possible, but definitely should not be the default
behavior. ( {{=true}} or something similarly clear
and not accident prone should be required) It is very cumbersome to delete a
large number of collections created by a routed alias right now. Not a common
problem in normal scenarios, but a potential PITA when configuration mistakes
are made.
> Ability to query aliases in Solr Admin UI
> -
>
> Key: SOLR-13122
> URL: https://issues.apache.org/jira/browse/SOLR-13122
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI
>Reporter: mosh
>Assignee: Jan Høydahl
>Priority: Major
> Labels: UI
> Fix For: 8.2
>
> Attachments: alias-collection-menu-selected.png,
> alias-collection-view.png, alias-collections-menu.png,
> alias-collections-menu.png, alias-delete-dialogue.png, alias-dropdown.png,
> alias-select-double.png, alias-view.png, new-collection-dropdown.png,
> new-oll-overview.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After having recently toyed with Time Routed Alias in SolrCloud,
> we have noticed there is no way to query an alias from the admin UI,
> since the combo box only contains the current collection in the cluster.
> Solr Admin UI ought to have a way to query these aliases, for better
> convenience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[ https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874144#comment-16874144 ] Gus Heck commented on SOLR-13122: - Yeah, that would definitely be scope creep for this ticket. That makes sense. > Ability to query aliases in Solr Admin UI > - > > Key: SOLR-13122 > URL: https://issues.apache.org/jira/browse/SOLR-13122 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Reporter: mosh >Assignee: Jan Høydahl >Priority: Major > Labels: UI > Fix For: 8.2 > > Attachments: alias-collection-menu-selected.png, > alias-collection-view.png, alias-collections-menu.png, > alias-collections-menu.png, alias-delete-dialogue.png, alias-dropdown.png, > alias-select-double.png, alias-view.png, new-collection-dropdown.png, > new-oll-overview.png > > Time Spent: 10m > Remaining Estimate: 0h > > After having recently toyed with Time Routed Alias in SolrCloud, > we have noticed there is no way to query an alias from the admin UI, > since the combo box only contains the current collection in the cluster. > Solr Admin UI ought to have a way to query these aliases, for better > convenience. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13122) Ability to query aliases in Solr Admin UI
[ https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874073#comment-16874073 ] Gus Heck edited comment on SOLR-13122 at 6/27/19 12:27 PM: --- I actually don't much like the collection drop down on the left. If anything goes it should be that. It's terribly hard to use if you have >10 collections or collection names longer than "foobar" Make one Time Routed Alias with daily collections for a month (which soon creates 30 collections named tweetsOrSomething_2019-06-XX) and then try to find your other collection "foobar" among the 30 wrapping entries in that drop down ... yeah you have type ahead, but the drop down part becomes entirely useless. If you don't know/can't remember the collection name (or are exploring a new client's install on a testing machine?) the drop down is a pretty bad view of what exists. (and you would never even find the aliases without a lot of looking). One convenience for testing what things look like with lots of collections would be to create a TRA with a start date in the past and set maxFutureMs to something very high, and then send in one doc (via documents) with an id and the current date... it will then create a bunch of longish collection names for you. Note that I once did this with monthly collections since 1970 (a client was trying and luckily failing to do that, I did it just for fun locally to see if it could even work) and it crushed the system and died after the 500th collection and like 20 minutes so don't get too carried away :). was (Author: gus_heck): I actually don't much like the collection drop down on the left. If anything goes it should be that. It's terribly hard to use if you have >10 collections or collection names longer than "foobar" Make one Time Routed Alias with daily collections for a month (which soon creates 30 collections named tweetsOrSomething_2019-06-XX) and then try to find you other collection "foobar" among the 30 wrapping entries in that drop down ... yeah you have type ahead, but the drop down part becomes entirely useless. If you don't know/can't remember the collection name (or are exploring a new client's install on a testing machine?) the drop down is a pretty bad view of what exists. (and you would never even find the aliases without a lot of looking). One convenience for testing what things look like with lots of collections would be to create a TRA with a start date in the past and set maxFutureMs to something very high, and then send in one doc (via documents) with an id and the current date... it will then create a bunch of longish collection names for you. Note that I once did this with monthly collections since 1970 (a client was trying and luckily failing to do that, I did it just for fun locally to see if it could even work) and it crushed the system and died after the 500th collection and like 20 minutes so don't get too carried away :). > Ability to query aliases in Solr Admin UI > - > > Key: SOLR-13122 > URL: https://issues.apache.org/jira/browse/SOLR-13122 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Reporter: mosh >Assignee: Jan Høydahl >Priority: Major > Labels: UI > Fix For: 8.2 > > Attachments: alias-collection-menu-selected.png, > alias-collection-view.png, alias-collections-menu.png, > alias-collections-menu.png, alias-delete-dialogue.png, alias-dropdown.png, > alias-select-double.png, alias-view.png, new-collection-dropdown.png, > new-oll-overview.png > > Time Spent: 10m > Remaining Estimate: 0h > > After having recently toyed with Time Routed Alias in SolrCloud, > we have noticed there is no way to query an alias from the admin UI, > since the combo box only contains the current collection in the cluster. > Solr Admin UI ought to have a way to query these aliases, for better > convenience. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[ https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16874073#comment-16874073 ] Gus Heck commented on SOLR-13122: - I actually don't much like the collection drop down on the left. If anything goes it should be that. It's terribly hard to use if you have >10 collections or collection names longer than "foobar" Make one Time Routed Alias with daily collections for a month (which soon creates 30 collections named tweetsOrSomething_2019-06-XX) and then try to find you other collection "foobar" among the 30 wrapping entries in that drop down ... yeah you have type ahead, but the drop down part becomes entirely useless. If you don't know/can't remember the collection name (or are exploring a new client's install on a testing machine?) the drop down is a pretty bad view of what exists. (and you would never even find the aliases without a lot of looking). One convenience for testing what things look like with lots of collections would be to create a TRA with a start date in the past and set maxFutureMs to something very high, and then send in one doc (via documents) with an id and the current date... it will then create a bunch of longish collection names for you. Note that I once did this with monthly collections since 1970 (a client was trying and luckily failing to do that, I did it just for fun locally to see if it could even work) and it crushed the system and died after the 500th collection and like 20 minutes so don't get too carried away :). > Ability to query aliases in Solr Admin UI > - > > Key: SOLR-13122 > URL: https://issues.apache.org/jira/browse/SOLR-13122 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Reporter: mosh >Assignee: Jan Høydahl >Priority: Major > Labels: UI > Fix For: 8.2 > > Attachments: alias-collection-menu-selected.png, > alias-collection-view.png, alias-collections-menu.png, > alias-collections-menu.png, alias-delete-dialogue.png, alias-dropdown.png, > alias-select-double.png, alias-view.png, new-collection-dropdown.png, > new-oll-overview.png > > Time Spent: 10m > Remaining Estimate: 0h > > After having recently toyed with Time Routed Alias in SolrCloud, > we have noticed there is no way to query an alias from the admin UI, > since the combo box only contains the current collection in the cluster. > Solr Admin UI ought to have a way to query these aliases, for better > convenience. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[ https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873675#comment-16873675 ] Gus Heck commented on SOLR-13122: - I mean remove the create/delete alias and all references to aliases from the collections page, and add an Aliases Item on the left, then provide a similar UI with aliases listed on the left, and details for the selected alias on the right, including a list of collections which may be clicked to go to their corresponding entry on the collections page. That would keep things clean consistent and we wouldn't have alias buttons floating along with details about specific collections that may or may not be involved in an alias. > Ability to query aliases in Solr Admin UI > - > > Key: SOLR-13122 > URL: https://issues.apache.org/jira/browse/SOLR-13122 > Project: Solr > Issue Type: Improvement > Components: Admin UI >Reporter: mosh >Assignee: Jan Høydahl >Priority: Major > Labels: UI > Fix For: 8.2 > > Attachments: alias-collection-menu-selected.png, > alias-collections-menu.png, alias-collections-menu.png, alias-dropdown.png, > alias-select-double.png > > Time Spent: 10m > Remaining Estimate: 0h > > After having recently toyed with Time Routed Alias in SolrCloud, > we have noticed there is no way to query an alias from the admin UI, > since the combo box only contains the current collection in the cluster. > Solr Admin UI ought to have a way to query these aliases, for better > convenience. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13122) Ability to query aliases in Solr Admin UI
[
https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873601#comment-16873601
]
Gus Heck edited comment on SOLR-13122 at 6/26/19 7:17 PM:
--
Looks nice, certainly an improvement. However I tend to think Aliases should
have it's own section, eventually it would be nice if alias creation expanded
to include routed aliases
Yes RA's metadata isreturned by LISTALIASES (below being what happens to be
returned from my local test solr while working on SOLR-13375.
{code:java}
{
"responseHeader": {
"status": 0,
"QTime": 2
},
"aliases": {
"dra_test":
"dra_test_CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_TRA__2019-01-01"
},
"properties": {
"dra_test": {
"router.0.field": "cat",
"router.0.maxCardinality": "20",
"router.1.field": "created_tdt",
"router.1.name": "time",
"router.field": "cat,created_tdt",
"name": "dra_test",
"router.name": "Dimensional[category,time]",
"router.1.start": "2019-01-01T00:00:00Z/MONTH",
"router.0.name": "category",
"router.1.interval": "+1MONTH",
"router.1.maxFutureMs": "31556952000",
"create-collection.collection.configName": "_default",
"create-collection.numShards": "2",
"operation": "createalias"
}
}
}
{code}
was (Author: gus_heck):
Looks nice, certainly an improvement. However I tend to think Aliases should
have it's own section, eventually it would be nice if alias creation expanded
to include routed aliases
Yes RA's metadata isreturned by LISTALIASES (below being what happens to be
what I get from local test solr while working on SOLR-13375.
{code:java}
{
"responseHeader": {
"status": 0,
"QTime": 2
},
"aliases": {
"dra_test":
"dra_test_CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_TRA__2019-01-01"
},
"properties": {
"dra_test": {
"router.0.field": "cat",
"router.0.maxCardinality": "20",
"router.1.field": "created_tdt",
"router.1.name": "time",
"router.field": "cat,created_tdt",
"name": "dra_test",
"router.name": "Dimensional[category,time]",
"router.1.start": "2019-01-01T00:00:00Z/MONTH",
"router.0.name": "category",
"router.1.interval": "+1MONTH",
"router.1.maxFutureMs": "31556952000",
"create-collection.collection.configName": "_default",
"create-collection.numShards": "2",
"operation": "createalias"
}
}
}
{code}
> Ability to query aliases in Solr Admin UI
> -
>
> Key: SOLR-13122
> URL: https://issues.apache.org/jira/browse/SOLR-13122
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI
>Reporter: mosh
>Assignee: Jan Høydahl
>Priority: Major
> Labels: UI
> Fix For: 8.2
>
> Attachments: alias-collection-menu-selected.png,
> alias-collections-menu.png, alias-collections-menu.png, alias-dropdown.png,
> alias-select-double.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After having recently toyed with Time Routed Alias in SolrCloud,
> we have noticed there is no way to query an alias from the admin UI,
> since the combo box only contains the current collection in the cluster.
> Solr Admin UI ought to have a way to query these aliases, for better
> convenience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[
https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873601#comment-16873601
]
Gus Heck commented on SOLR-13122:
-
Looks nice, certainly an improvement. However I tend to think Aliases should
have it's own section, eventually it would be nice if alias creation expanded
to include routed aliases
Yes RA's metadata isreturned by LISTALIASES (below being what happens to be
what I get from local test solr while working on SOLR-13375.
{code:java}
{
"responseHeader": {
"status": 0,
"QTime": 2
},
"aliases": {
"dra_test":
"dra_test_CRA__NEW_CATEGORY_ROUTED_ALIAS_WAITING_FOR_DATA_TEMP_TRA__2019-01-01"
},
"properties": {
"dra_test": {
"router.0.field": "cat",
"router.0.maxCardinality": "20",
"router.1.field": "created_tdt",
"router.1.name": "time",
"router.field": "cat,created_tdt",
"name": "dra_test",
"router.name": "Dimensional[category,time]",
"router.1.start": "2019-01-01T00:00:00Z/MONTH",
"router.0.name": "category",
"router.1.interval": "+1MONTH",
"router.1.maxFutureMs": "31556952000",
"create-collection.collection.configName": "_default",
"create-collection.numShards": "2",
"operation": "createalias"
}
}
}
{code}
> Ability to query aliases in Solr Admin UI
> -
>
> Key: SOLR-13122
> URL: https://issues.apache.org/jira/browse/SOLR-13122
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI
>Reporter: mosh
>Assignee: Jan Høydahl
>Priority: Major
> Labels: UI
> Fix For: 8.2
>
> Attachments: alias-collection-menu-selected.png,
> alias-collections-menu.png, alias-collections-menu.png, alias-dropdown.png,
> alias-select-double.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> After having recently toyed with Time Routed Alias in SolrCloud,
> we have noticed there is no way to query an alias from the admin UI,
> since the combo box only contains the current collection in the cluster.
> Solr Admin UI ought to have a way to query these aliases, for better
> convenience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872310#comment-16872310 ] Gus Heck commented on SOLR-13375: - Actually looking again this morning I realize that the writeParams method is a bit of a red herring. It's really the conflict between the getParams() method api and the objects I want to express. The real problem is that the v2 API (as I am attempting to use it) wants to be able to handle more complex objects than SolrParams really was intended for. SolrParams *is* documented as being string to one or more strings, but that makes it hard to handle json that has properties that are lists of objects (lists of strings clearly work). Autoscaling seems to be using lists of object for set-trigger.actions but AFAICT they don't have a v1 api and I suspect they therefore dodge this SolrParams.toMap()/getParam() issue. One possible way around this might be to override toMap() in the wrapper to just return the map that backs the wrapper, but that has to still somehow do the conversions to v1 api keys before returning the map, and it widens the actual capabilities of SolrParams beyond it's documentation which could trip up other folks. > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872310#comment-16872310 ] Gus Heck edited comment on SOLR-13375 at 6/25/19 12:31 PM: --- Actually looking again this morning I realize that the writeMap method is a bit of a red herring. It's really the conflict between the getParams() method api and the objects I want to express. The real problem is that the v2 API (as I am attempting to use it) wants to be able to handle more complex objects than SolrParams really was intended for. SolrParams *is* documented as being string to one or more strings, but that makes it hard to handle json that has properties that are lists of objects (lists of strings clearly work). Autoscaling seems to be using lists of object for set-trigger.actions but AFAICT they don't have a v1 api and I suspect they therefore dodge this SolrParams.toMap()/getParam() issue. One possible way around this might be to override toMap() in the wrapper to just return the map that backs the wrapper, but that has to still somehow do the conversions to v1 api keys before returning the map, and it widens the actual capabilities of SolrParams beyond it's documentation which could trip up other folks. was (Author: gus_heck): Actually looking again this morning I realize that the writeParams method is a bit of a red herring. It's really the conflict between the getParams() method api and the objects I want to express. The real problem is that the v2 API (as I am attempting to use it) wants to be able to handle more complex objects than SolrParams really was intended for. SolrParams *is* documented as being string to one or more strings, but that makes it hard to handle json that has properties that are lists of objects (lists of strings clearly work). Autoscaling seems to be using lists of object for set-trigger.actions but AFAICT they don't have a v1 api and I suspect they therefore dodge this SolrParams.toMap()/getParam() issue. One possible way around this might be to override toMap() in the wrapper to just return the map that backs the wrapper, but that has to still somehow do the conversions to v1 api keys before returning the map, and it widens the actual capabilities of SolrParams beyond it's documentation which could trip up other folks. > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16872301#comment-16872301 ] Gus Heck commented on SOLR-13375: - org.apache.solr.handler.admin.BaseHandlerApiSupport#wrapParams > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13375) Dimensional Routed Aliases
[ https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13375: Attachment: SOLR-13375.patch > Dimensional Routed Aliases > -- > > Key: SOLR-13375 > URL: https://issues.apache.org/jira/browse/SOLR-13375 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13375.patch > > > Current available routed aliases are restricted to a single field. This > feature will allow Solr to provide data driven collection access, creation > and management based on multiple fields in a document. The collections will > be queried and updated in a unified manner via an alias. Current routing is > restricted to the values of a single field. The particularly useful > combination at this time will be Category X Time routing but Category X > Category may also be useful. More importantly, if additional routing schemes > are created in the future (either as contributions or as custom code by > users) combination among these should be supported. > It is expected that not all combinations will be useful, and that > determination of usefulness I expect to leave up to the user. Some Routing > schemes may need to be limited to be the leaf/last routing scheme for > technical reasons, though I'm not entirely convinced of that yet. If so, a > flag will be added to the RoutedAlias interface. > Initial desire is to support two levels, though if arbitrary levels can be > supported easily that will be done. > This could also have been called CompositeRoutedAlias, but that creates a TLA > clash with CategoryRoutedAlias. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13375) Dimensional Routed Aliases
[
https://issues.apache.org/jira/browse/SOLR-13375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871936#comment-16871936
]
Gus Heck commented on SOLR-13375:
-
Attaching patch with WIP initial concept of the V1 api for this working to the
point of creating an alias that thinks it's a DRA (but fails with a not yet
implemented exception message if you try to send it data) API looks like this:
{code:java}
http://localhost:8983/solr/admin/collections?action=CREATEALIAS
=dra_test
=Dimensional[category,time]
=myCategory_s
=20
=myDate_tdt
=2019-01-01T00:00:00Z/MONTH
=%2B1MONTH
=60
=_default
=2
{code}
This is not mapped into the V2 API yet because although I want to do this:
{code:java}
"routerList": {
"type": "array",
"description": "A list of router property sets to be used with
router type Dimensional[foo,bar] where foo and bar are valid router type names
(i.e. time or category). The order must correspond to the type specification in
[] in the Dimensional type, so Dimensional[category,time] would require the
first set of router properties to be valid for a category routed alias, and the
second set to be valid for a time routed alias. In these sets of properties,
router.name will be ignored in favor of the type specified in the top level
Dimensional[] router.name",
"items": {
"type": "object",
"additionalProperties": true
}
}
{code}
enabling this v2 api JSON:
{code:java}
{
"create-alias":{
"name":"dra_test2",
"router": {
"name": "Dimensional[category,time]",
"routerList" : [{
"field":"myCategory_s",
"maxCardinality":20
}, {
"field":"myDate_tdt",
"start":"2019-01-01T00:00:00Z",
"interval":"+1MONTH",
"maxFutureMs":60
}]
},
"create-collection": {
"collection.configName":"_default",
"numShards":2
}
}
}
{code}
this todo/assumption from SOLR-11913 is getting in the way ([~dsmiley]):
{code:java}
public void writeMap(EntryWriter ew) throws IOException {
//TODO don't call toNamedList; more efficiently implement here
//note: multiple values, if present, are a String[] under 1 key
toNamedList().forEach((k, v) -> {
{code}
And throwing:
{code:java}
"error": {
"metadata": [
"error-class",
"org.apache.solr.common.SolrException",
"root-error-class",
"java.lang.ArrayStoreException"
],
"msg": "java.lang.ArrayStoreException: arraycopy: element type
mismatch: can not cast one of the elements of java.lang.Object[] to the type of
the destination array, java.lang.String",
"code": 400
}
{code}
The reason for this is that my configuration results in a SolrParams wrapper
instance that has non-string (List) values in the map variable ( which carried
along by the lambda as backing for getParams(), which returns String[] and uses
List.toArray() with a String array parameter)... I may be able to work around
this by not using toMap() but that's probably going to be messier
> Dimensional Routed Aliases
> --
>
> Key: SOLR-13375
> URL: https://issues.apache.org/jira/browse/SOLR-13375
> Project: Solr
> Issue Type: New Feature
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
>
> Current available routed aliases are restricted to a single field. This
> feature will allow Solr to provide data driven collection access, creation
> and management based on multiple fields in a document. The collections will
> be queried and updated in a unified manner via an alias. Current routing is
> restricted to the values of a single field. The particularly useful
> combination at this time will be Category X Time routing but Category X
> Category may also be useful. More importantly, if additional routing schemes
> are created in the future (either as contributions or as custom code by
> users) combination among these should be supported.
> It is expected that not all combinations will be useful, and that
> determination of usefulness I expect to leave up to the user. Some Routing
> schemes may need to be limited to be the leaf/last routing scheme for
> technical reasons, though I'm not entirely convinced of that yet. If so, a
> flag will be added to the RoutedAlias interface.
> Initial desire is to
[jira] [Comment Edited] (SOLR-13122) Ability to query aliases in Solr Admin UI
[
https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871652#comment-16871652
]
Gus Heck edited comment on SOLR-13122 at 6/24/19 6:26 PM:
--
{quote}
Is it so now that every collection created now has an alias with the same name
pointing to it? If so, perhaps skip these aliases in the list?
{quote}
Not unless I've missed a recent change
{quote}
Would the schema endpoint work for aliases? Other endpoints that make sense to
show for aliases
{quote}
[~ab] recently made a change to make aliases that refer to single collections
work with most commands, so I expect so but he probably knows for sure
{quote}
Should we make any special treatment of an alias X that point to exactly one
collection Y? Versus those that point to multiple collections? For clarity I
think treating all aliases the same and not trying to be smart routing some
requests to the alias and some to the underlying collection is the best choice
{quote}
As noted in my last comment multi-collection and routed aliases are not
supported in many commands, because the logic becomes unclear so you probably
do need to handle them differently.
Documents can be enabled for single collection aliases and routed aliases to
since you can send updates to those types of aliases.
was (Author: gus_heck):
{quote}
Is it so now that every collection created now has an alias with the same name
pointing to it? If so, perhaps skip these aliases in the list?
{quote}
Not unless I've missed a recent change
{quote}
Would the schema endpoint work for aliases? Other endpoints that make sense to
show for aliases
{quote}
[~a...@getopt.org] recently made a change to make aliases that refer to single
collections work with most commands, so I expect so but he probably knows for
sure
{quote}
Should we make any special treatment of an alias X that point to exactly one
collection Y? Versus those that point to multiple collections? For clarity I
think treating all aliases the same and not trying to be smart routing some
requests to the alias and some to the underlying collection is the best choice
{quote}
As noted in my last comment multi-collection and routed aliases are not
supported in many commands, because the logic becomes unclear so you probably
do need to handle them differently.
Documents can be enabled for single collection aliases and routed aliases to
since you can send updates to those types of aliases.
> Ability to query aliases in Solr Admin UI
> -
>
> Key: SOLR-13122
> URL: https://issues.apache.org/jira/browse/SOLR-13122
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI
>Reporter: mosh
>Assignee: Jan Høydahl
>Priority: Major
> Labels: UI
>
> After having recently toyed with Time Routed Alias in SolrCloud,
> we have noticed there is no way to query an alias from the admin UI,
> since the combo box only contains the current collection in the cluster.
> Solr Admin UI ought to have a way to query these aliases, for better
> convenience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13122) Ability to query aliases in Solr Admin UI
[
https://issues.apache.org/jira/browse/SOLR-13122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16871652#comment-16871652
]
Gus Heck commented on SOLR-13122:
-
{quote}
Is it so now that every collection created now has an alias with the same name
pointing to it? If so, perhaps skip these aliases in the list?
{quote}
Not unless I've missed a recent change
{quote}
Would the schema endpoint work for aliases? Other endpoints that make sense to
show for aliases
{quote}
[~a...@getopt.org] recently made a change to make aliases that refer to single
collections work with most commands, so I expect so but he probably knows for
sure
{quote}
Should we make any special treatment of an alias X that point to exactly one
collection Y? Versus those that point to multiple collections? For clarity I
think treating all aliases the same and not trying to be smart routing some
requests to the alias and some to the underlying collection is the best choice
{quote}
As noted in my last comment multi-collection and routed aliases are not
supported in many commands, because the logic becomes unclear so you probably
do need to handle them differently.
Documents can be enabled for single collection aliases and routed aliases to
since you can send updates to those types of aliases.
> Ability to query aliases in Solr Admin UI
> -
>
> Key: SOLR-13122
> URL: https://issues.apache.org/jira/browse/SOLR-13122
> Project: Solr
> Issue Type: Improvement
> Components: Admin UI
>Reporter: mosh
>Assignee: Jan Høydahl
>Priority: Major
> Labels: UI
>
> After having recently toyed with Time Routed Alias in SolrCloud,
> we have noticed there is no way to query an alias from the admin UI,
> since the combo box only contains the current collection in the cluster.
> Solr Admin UI ought to have a way to query these aliases, for better
> convenience.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13419) Add TRA denoting Infix to TRA collection names
[ https://issues.apache.org/jira/browse/SOLR-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13419. - Resolution: Fixed Fix Version/s: 8.2 > Add TRA denoting Infix to TRA collection names > -- > > Key: SOLR-13419 > URL: https://issues.apache.org/jira/browse/SOLR-13419 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Fix For: 8.2 > > Attachments: SOLR-13419.patch, SOLR-13419.patch > > > One of the things that Dimensional Routed aliases (DRA) will need to be able > to do is provide the component routing features based on the route value in > the collection name. With multiple dimensions it becomes necessary to pull > this information out on a per-dimension basis and the current Time Routed > Alias (TRA) collection names have only a simple underscore between the alias > name and the route value timestamp. Furthermore the timestamps are variable > in length, having truncation of low significance zero fields as an > intentional feature (collections falling evenly on the month boundary will > omit the day/hour/min fields). This makes recognizing the TRA part of a DRA > complex. This issue will bring TRA's in line with CRA's so that both follow > the pattern: > alias__type__routevalue > Specifically the new pattern will use __TRA__ for the type section. This > change must not break existing TRA's, and will not update existing collection > names, but the two formats have to co-exist peacefully going forward. DRA's > will only support time dimensions with the new format however, with a general > form of: > alias__type__routevalue__type__routevalue -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-8872) upgrade ecj on branch_8x to get bug fixes for linter
[ https://issues.apache.org/jira/browse/LUCENE-8872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868764#comment-16868764 ] Gus Heck commented on LUCENE-8872: -- +1 > upgrade ecj on branch_8x to get bug fixes for linter > > > Key: LUCENE-8872 > URL: https://issues.apache.org/jira/browse/LUCENE-8872 > Project: Lucene - Core > Issue Type: Task >Reporter: Hoss Man >Assignee: Hoss Man >Priority: Major > Attachments: LUCENE-8872.patch > > > SOLR-13534 recently triggered an NPE in the 'ecj-lint' task on branch_8x. > Looking into it, it seems like this is the bug.. > https://bugs.eclipse.org/bugs/show_bug.cgi?id=502871 > ...upgrading ECJ should fix this -- confirmed by some quick local testing i > just did. > Uwe actually upgraded ECJ on master already as part of the change to java11, > but the smaller portion of just upgrading ECJ wasn't backported -- there > doesn't seem to be any reason *NOT* to backport just he ECJ org/revision > change (w/o chnaging the supported version) > https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=c1d0889 -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13534) Dynamic loading of jars from a url
[
https://issues.apache.org/jira/browse/SOLR-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868744#comment-16868744
]
Gus Heck commented on SOLR-13534:
-
changing
{code:java}
return getBlobIncRef(keyBuilder.toString(), () -> new BlobContent<>(key,
fetchBlobAndVerify(key, url, sha512), decoder));
{code}
to
{code:java}
return getBlobIncRef(keyBuilder.toString(), () -> new BlobContent(key,
fetchBlobAndVerify(key, url, sha512), decoder)); {code}
works around the problem but is your patch in LUCENE-8872 rolling back the ecj
version a better fix?
> Dynamic loading of jars from a url
> --
>
> Key: SOLR-13534
> URL: https://issues.apache.org/jira/browse/SOLR-13534
> Project: Solr
> Issue Type: Improvement
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Dynamic loading is possible from {{.system}} collection. It's much easier to
> host the jars on a remote service and load it from there. This way the user
> should have no problem in loading jars when the {{.system}} collection is not
> available for some reason.
> The steps should look as follows
> # get the hash of your jar file. {{openssl dgst -sha512 }}
> # upload it your hosting service . say the location is
> {{[http://host:port/my-jar/location|http://hostport/]}}
> # create a runtime lib entry for the collection as follows
> {code:java}
> curl http://localhost:8983/solr/techproducts/config -H
> 'Content-type:application/json' -d '{
>"add-runtimelib": { "name":"jarblobname",
> "sha512":"e94bb3990b39aacdabaa3eef7ca6102d96fa46766048da50269f25fd41164440a4e024d7a7fb0d5ec328cd8322bb65f5ba7886e076a8f224f78cb310fd45896d"
> , "url" : "http://host:port/my-jar/loaction"}
> }'
> {code}
> to update the jar, just repeat the steps and use the {{update-runtimelib}} to
> update the sha512 hash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13534) Dynamic loading of jars from a url
[
https://issues.apache.org/jira/browse/SOLR-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868656#comment-16868656
]
Gus Heck edited comment on SOLR-13534 at 6/20/19 4:03 PM:
--
This appears to be failing precommit on 8x. When I roll this back it succeeds.
[https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/185/]
I hit it locally on ubuntu.
was (Author: gus_heck):
This appears to be failing precommit on 8x. When I roll this back it succeeds.
[https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/185/]
I hit it locally too on ubuntu.
> Dynamic loading of jars from a url
> --
>
> Key: SOLR-13534
> URL: https://issues.apache.org/jira/browse/SOLR-13534
> Project: Solr
> Issue Type: Improvement
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Dynamic loading is possible from {{.system}} collection. It's much easier to
> host the jars on a remote service and load it from there. This way the user
> should have no problem in loading jars when the {{.system}} collection is not
> available for some reason.
> The steps should look as follows
> # get the hash of your jar file. {{openssl dgst -sha512 }}
> # upload it your hosting service . say the location is
> {{[http://host:port/my-jar/location|http://hostport/]}}
> # create a runtime lib entry for the collection as follows
> {code:java}
> curl http://localhost:8983/solr/techproducts/config -H
> 'Content-type:application/json' -d '{
>"add-runtimelib": { "name":"jarblobname",
> "sha512":"e94bb3990b39aacdabaa3eef7ca6102d96fa46766048da50269f25fd41164440a4e024d7a7fb0d5ec328cd8322bb65f5ba7886e076a8f224f78cb310fd45896d"
> , "url" : "http://host:port/my-jar/loaction"}
> }'
> {code}
> to update the jar, just repeat the steps and use the {{update-runtimelib}} to
> update the sha512 hash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13534) Dynamic loading of jars from a url
[
https://issues.apache.org/jira/browse/SOLR-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868656#comment-16868656
]
Gus Heck commented on SOLR-13534:
-
This appears to be failing precommit on 8x. When I roll this back it succeeds.
[https://jenkins.thetaphi.de/job/Lucene-Solr-8.x-Solaris/185/]
I hit it locally too on ubuntu.
> Dynamic loading of jars from a url
> --
>
> Key: SOLR-13534
> URL: https://issues.apache.org/jira/browse/SOLR-13534
> Project: Solr
> Issue Type: Improvement
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.2
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Dynamic loading is possible from {{.system}} collection. It's much easier to
> host the jars on a remote service and load it from there. This way the user
> should have no problem in loading jars when the {{.system}} collection is not
> available for some reason.
> The steps should look as follows
> # get the hash of your jar file. {{openssl dgst -sha512 }}
> # upload it your hosting service . say the location is
> {{[http://host:port/my-jar/location|http://hostport/]}}
> # create a runtime lib entry for the collection as follows
> {code:java}
> curl http://localhost:8983/solr/techproducts/config -H
> 'Content-type:application/json' -d '{
>"add-runtimelib": { "name":"jarblobname",
> "sha512":"e94bb3990b39aacdabaa3eef7ca6102d96fa46766048da50269f25fd41164440a4e024d7a7fb0d5ec328cd8322bb65f5ba7886e076a8f224f78cb310fd45896d"
> , "url" : "http://host:port/my-jar/loaction"}
> }'
> {code}
> to update the jar, just repeat the steps and use the {{update-runtimelib}} to
> update the sha512 hash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
[ https://issues.apache.org/jira/browse/SOLR-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867845#comment-16867845 ] Gus Heck commented on SOLR-13559: - I also tried beasting 100 times with just one runner no joy there either. (or rather all joy and no fail). Yes the code SAYS it waits for this to exist, but evidently it dosn't... it's probably a zk propgation or something funky with the caching in BaseHttpClusterStateProvider... I do notice that it does getAliases(false); in getAliasProperties(String alias) but I haven't been able to convince myself as to why that would only sometimes be a problem... > AliasIntegrationTest.testClusterStateProviderAPI fails to often > --- > > Key: SOLR-13559 > URL: https://issues.apache.org/jira/browse/SOLR-13559 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: Tests >Affects Versions: master (9.0) >Reporter: Gus Heck >Priority: Minor > > Recent failure rates for AliasIntegrationTest.testClusterStateProviderAPI > have been around 4% which is too high. > (http://fucit.org/solr-jenkins-reports/failure-report.html). I've beasted 100 > runs a couple times and not reproduced it but then hit a failure in it during > a normal test run today, so I'm going to start this ticket and record the > trace. I have yet to dig through the zips for the logs from the builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-12357) TRA: Pre-emptively create next collection
[ https://issues.apache.org/jira/browse/SOLR-12357?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867837#comment-16867837 ] Gus Heck commented on SOLR-12357: - These are interesting possibilities and I do have ideas here, but these should be discussed on the dev list and/or in their own ticket. > TRA: Pre-emptively create next collection > -- > > Key: SOLR-12357 > URL: https://issues.apache.org/jira/browse/SOLR-12357 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Reporter: David Smiley >Assignee: David Smiley >Priority: Major > Fix For: 7.5 > > Attachments: SOLR-12357.patch > > Time Spent: 9.5h > Remaining Estimate: 0h > > When adding data to a Time Routed Alias (TRA), we sometimes need to create > new collections. Today we only do this synchronously – on-demand when a > document is coming in. But this can add delays as the documents inbound are > held up for a collection to be created. And, there may be a problem like a > lack of resources (e.g. ample SolrCloud nodes with space) that the policy > framework defines. Such problems could be rectified sooner rather than later > assume there is log alerting in place (definitely out of scope here). > Pre-emptive TRA collection needs a time window configuration parameter, > perhaps named something like "preemptiveCreateWindowMs". If a document's > timestamp is within this time window _from the end time of the head/lead > collection_ then the collection can be created pre-eptively. If no data is > being sent to the TRA, no collections will be auto created, nor will it > happen if older data is being added. It may be convenient to effectively > limit this time setting to the _smaller_ of this value and the TRA interval > window, which I think is a fine limitation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
[
https://issues.apache.org/jira/browse/SOLR-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866811#comment-16866811
]
Gus Heck edited comment on SOLR-13559 at 6/18/19 4:32 PM:
--
Note: this seed does not reproduce, the {{& gt;}} etc is because I pulled this
from the xml file from the test output.
{code:java}
java.lang.AssertionError: {} expected:2 but was:0
at
__randomizedtesting.SeedInfo.seed([DBB80F0F76EA5A34:C46F932305E1A37F]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.solr.cloud.AliasIntegrationTest.testClusterStateProviderAPI(AliasIntegrationTest.java:303)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
was (Author: gus_heck):
Note: this seed does not reproduce, the etc is because I pulled this from
the xml file from
[jira] [Commented] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
[
https://issues.apache.org/jira/browse/SOLR-13559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866811#comment-16866811
]
Gus Heck commented on SOLR-13559:
-
Note: this seed does not reproduce, the etc is because I pulled this from
the xml file from the test output.
{code:java}
java.lang.AssertionError: {} expected:2 but was:0
at
__randomizedtesting.SeedInfo.seed([DBB80F0F76EA5A34:C46F932305E1A37F]:0)
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at
org.apache.solr.cloud.AliasIntegrationTest.testClusterStateProviderAPI(AliasIntegrationTest.java:303)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1750)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:938)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:974)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:988)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:49)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
org.apache.lucene.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:48)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:817)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:468)
at
com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:947)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:832)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:883)
at
com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:894)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.SystemPropertiesRestoreRule$1.evaluate(SystemPropertiesRestoreRule.java:57)
at
org.apache.lucene.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:45)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:41)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
org.apache.lucene.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
at
org.apache.lucene.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:47)
at
org.apache.lucene.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:64)
at
org.apache.lucene.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:54)
at
com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
at
com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:368)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}
> AliasIntegrationTest.testClusterStateProviderAPI fails to often
> ---
>
> Key: SOLR-13559
>
[jira] [Created] (SOLR-13559) AliasIntegrationTest.testClusterStateProviderAPI fails to often
Gus Heck created SOLR-13559: --- Summary: AliasIntegrationTest.testClusterStateProviderAPI fails to often Key: SOLR-13559 URL: https://issues.apache.org/jira/browse/SOLR-13559 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: Tests Affects Versions: master (9.0) Reporter: Gus Heck Recent failure rates for AliasIntegrationTest.testClusterStateProviderAPI have been around 4% which is too high. (http://fucit.org/solr-jenkins-reports/failure-report.html). I've beasted 100 runs a couple times and not reproduced it but then hit a failure in it during a normal test run today, so I'm going to start this ticket and record the trace. I have yet to dig through the zips for the logs from the builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13419) Add TRA denoting Infix to TRA collection names
[ https://issues.apache.org/jira/browse/SOLR-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16865907#comment-16865907 ] Gus Heck commented on SOLR-13419: - Updated patch for current master, just some test readability changes and CHANGES.txt entry. Will commit soon since this has been avail for review for 1.5 mo. > Add TRA denoting Infix to TRA collection names > -- > > Key: SOLR-13419 > URL: https://issues.apache.org/jira/browse/SOLR-13419 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13419.patch, SOLR-13419.patch > > > One of the things that Dimensional Routed aliases (DRA) will need to be able > to do is provide the component routing features based on the route value in > the collection name. With multiple dimensions it becomes necessary to pull > this information out on a per-dimension basis and the current Time Routed > Alias (TRA) collection names have only a simple underscore between the alias > name and the route value timestamp. Furthermore the timestamps are variable > in length, having truncation of low significance zero fields as an > intentional feature (collections falling evenly on the month boundary will > omit the day/hour/min fields). This makes recognizing the TRA part of a DRA > complex. This issue will bring TRA's in line with CRA's so that both follow > the pattern: > alias__type__routevalue > Specifically the new pattern will use __TRA__ for the type section. This > change must not break existing TRA's, and will not update existing collection > names, but the two formats have to co-exist peacefully going forward. DRA's > will only support time dimensions with the new format however, with a general > form of: > alias__type__routevalue__type__routevalue -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13419) Add TRA denoting Infix to TRA collection names
[ https://issues.apache.org/jira/browse/SOLR-13419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13419: Attachment: SOLR-13419.patch > Add TRA denoting Infix to TRA collection names > -- > > Key: SOLR-13419 > URL: https://issues.apache.org/jira/browse/SOLR-13419 > Project: Solr > Issue Type: Sub-task > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13419.patch, SOLR-13419.patch > > > One of the things that Dimensional Routed aliases (DRA) will need to be able > to do is provide the component routing features based on the route value in > the collection name. With multiple dimensions it becomes necessary to pull > this information out on a per-dimension basis and the current Time Routed > Alias (TRA) collection names have only a simple underscore between the alias > name and the route value timestamp. Furthermore the timestamps are variable > in length, having truncation of low significance zero fields as an > intentional feature (collections falling evenly on the month boundary will > omit the day/hour/min fields). This makes recognizing the TRA part of a DRA > complex. This issue will bring TRA's in line with CRA's so that both follow > the pattern: > alias__type__routevalue > Specifically the new pattern will use __TRA__ for the type section. This > change must not break existing TRA's, and will not update existing collection > names, but the two formats have to co-exist peacefully going forward. DRA's > will only support time dimensions with the new format however, with a general > form of: > alias__type__routevalue__type__routevalue -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13551) Fix directional reference in aliases.adoc
[ https://issues.apache.org/jira/browse/SOLR-13551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13551. - Resolution: Fixed Fix Version/s: 8.2 > Fix directional reference in aliases.adoc > - > > Key: SOLR-13551 > URL: https://issues.apache.org/jira/browse/SOLR-13551 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: documentation >Affects Versions: 8.1 >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Trivial > Fix For: 8.2 > > > * an alias must not refer to a [Routed > Alias|https://lucene.apache.org/solr/guide/8_1/aliases.html#routed-aliases] > (see below) > occurs below the Routed Alias section. Removing the parenthetical since we > have a link already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13551) Fix directional reference in aliases.adoc
Gus Heck created SOLR-13551: --- Summary: Fix directional reference in aliases.adoc Key: SOLR-13551 URL: https://issues.apache.org/jira/browse/SOLR-13551 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Components: documentation Affects Versions: 8.1 Reporter: Gus Heck Assignee: Gus Heck * an alias must not refer to a [Routed Alias|https://lucene.apache.org/solr/guide/8_1/aliases.html#routed-aliases] (see below) occurs below the Routed Alias section. Removing the parenthetical since we have a link already. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13439: Fix Version/s: (was: master (9.0)) > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Fix For: 8.2 > > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13420) Allow Routed Aliases to use Collection Properties instead of core properties
[
https://issues.apache.org/jira/browse/SOLR-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-13420:
Fix Version/s: (was: master (9.0))
> Allow Routed Aliases to use Collection Properties instead of core properties
>
>
> Key: SOLR-13420
> URL: https://issues.apache.org/jira/browse/SOLR-13420
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Fix For: 8.2
>
> Attachments: SOLR-13420.patch, SOLR-13420.patch, SOLR-13420.patch,
> SOLR-13420.patch, SOLR-13420.patch
>
>
> The current routed alias code is relying on a core property named
> routedAliasName to detect when the Routed Alias wrapper URP should be applied
> to Distributed Update Request Processor.
> {code:java}
> #Written by CorePropertiesLocator
> #Sun Mar 03 06:21:14 UTC 2019
> routedAliasName=testalias21
> numShards=2
> collection.configName=_default
> ... etc...
> {code}
> Core properties are not changeable after the core is created, and they are
> written to the file system for every core. To support a unit test for
> SOLR-13419 I need to create some legacy formatted collection names, and
> arrange them into a TRA, but this is impossible due to the fact that I can't
> change the core property from the test. There's a TODO dating back to the
> original TRA implementation in the routed alias code to switch to collection
> properties instead, so this ticket will address that TODO to support the test
> required for SOLR-13419.
> Back compatibility with legacy core based TRA's and CRA's will of course be
> maintained. I also expect that this will facilitate some more nimble handling
> or routed aliases with future auto-scaling capabilities such as possibly
> detaching and archiving collections to cheaper, slower machines rather than
> deleting them. (presently such a collection would still attempt to use the
> routed alias if it received an update even if it were no longer in the list
> of collections for the alias)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13439. - Resolution: Fixed Fix Version/s: 8.2 master (9.0) > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Fix For: master (9.0), 8.2 > > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13420) Allow Routed Aliases to use Collection Properties instead of core properties
[
https://issues.apache.org/jira/browse/SOLR-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck resolved SOLR-13420.
-
Resolution: Fixed
Fix Version/s: 8.2
master (9.0)
> Allow Routed Aliases to use Collection Properties instead of core properties
>
>
> Key: SOLR-13420
> URL: https://issues.apache.org/jira/browse/SOLR-13420
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Fix For: master (9.0), 8.2
>
> Attachments: SOLR-13420.patch, SOLR-13420.patch, SOLR-13420.patch,
> SOLR-13420.patch, SOLR-13420.patch
>
>
> The current routed alias code is relying on a core property named
> routedAliasName to detect when the Routed Alias wrapper URP should be applied
> to Distributed Update Request Processor.
> {code:java}
> #Written by CorePropertiesLocator
> #Sun Mar 03 06:21:14 UTC 2019
> routedAliasName=testalias21
> numShards=2
> collection.configName=_default
> ... etc...
> {code}
> Core properties are not changeable after the core is created, and they are
> written to the file system for every core. To support a unit test for
> SOLR-13419 I need to create some legacy formatted collection names, and
> arrange them into a TRA, but this is impossible due to the fact that I can't
> change the core property from the test. There's a TODO dating back to the
> original TRA implementation in the routed alias code to switch to collection
> properties instead, so this ticket will address that TODO to support the test
> required for SOLR-13419.
> Back compatibility with legacy core based TRA's and CRA's will of course be
> maintained. I also expect that this will facilitate some more nimble handling
> or routed aliases with future auto-scaling capabilities such as possibly
> detaching and archiving collections to cheaper, slower machines rather than
> deleting them. (presently such a collection would still attempt to use the
> routed alias if it received an update even if it were no longer in the list
> of collections for the alias)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13544) Consider adding isCloud() method as shortcut for cloudDescriptor == null checks
[
https://issues.apache.org/jira/browse/SOLR-13544?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-13544:
Description:
In writing SOLR-13420 I was not supper happy with the verbosity of some of my
changes, and I think a convenience method could improve code legibility.
After doing some searching I found that in the first 50ish places where
getCloudDescriptor() is called. 8 of them are followed by a null check, which
is generally necessitated in cases where the code might run in non-cloud
environments. There are a great many places where null is not checked and cloud
is assumed, presumably for reasons related to the nature of the code (i.e.
cdcr, code in zkcontroller), but some code does check, and the ethos generally
seems to be "do x if we're in cloud".
For example in UpdateHandler I see
{code:java}
// If this is a replica of type PULL, don't create the update log
boolean skipUpdateLog = core.getCoreDescriptor().getCloudDescriptor() !=
null && !core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
This would be more concise and readable as:
{code:java}
boolean skipUpdateLog = core.isCloud() &&
!core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
Or in the routed alias code:
{code:java}
SolrCore core = req.getCore();
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
if (cloudDescriptor != null) {
String collectionName = cloudDescriptor.getCollectionName();
CoreContainer coreContainer = core.getCoreContainer();
ZkController zkController = coreContainer.getZkController();
... etc ...{code}
vs this which seems to read much smoother...
{code:java}
SolrCore core = req.getCore();
if (core.isCloud()) {
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
String collectionName = cloudDescriptor.getCollectionName();
... etc ...
{code}
The idea would be to add to SolrCore this method
{code:java}
public boolean isCloud() {
return getCoreDescriptor().getCloudDescriptor() != null;
} {code}
We also might add some asserts in there for other things folks might want to
assume to be true in the cloud case and javadoc what's reliable when this
returns true, but keep it skinny in production. The key question is are there
any cases where we're in solr cloud but getCloudDescriptor() is null? Seems
hard to imagine, but what do others think?
was:
In writing SOLR-13420 I was not supper happy with the verbosity of some of my
changes, and I think a convenience method could improve code legibility.
After doing some searching I found that in the first 50ish places where
getCloudDescriptor() is called. 8 of them are followed by a null check, which
is generally necessitated in cases where the code might run in non-cloud
environments. There are a great many places where null is not checked and cloud
is assumed, presumably for reasons related to the nature of the code (i.e.
cdcr, code in zkcontroller), but some code does check, and the ethos generally
seems to be "do x if we're in cloud".
For example in MDCLoggingContext.java I see:
{code:java}
// If this is a replica of type PULL, don't create the update log
boolean skipUpdateLog = core.getCoreDescriptor().getCloudDescriptor() !=
null && !core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
This would be more concise and readable as:
{code:java}
boolean skipUpdateLog = core.isCloud() &&
!core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
Or in the routed alias code:
{code:java}
SolrCore core = req.getCore();
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
if (cloudDescriptor != null) {
String collectionName = cloudDescriptor.getCollectionName();
CoreContainer coreContainer = core.getCoreContainer();
ZkController zkController = coreContainer.getZkController();
... etc ...{code}
vs this which seems to read much smoother...
{code:java}
SolrCore core = req.getCore();
if (core.isCloud()) {
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
String collectionName = cloudDescriptor.getCollectionName();
... etc ...
{code}
The idea would be to add to SolrCore this method
{code:java}
public boolean isCloud() {
return getCoreDescriptor().getCloudDescriptor() != null;
} {code}
We also might add some asserts in there for other things folks might want to
assume to be true in the cloud case and javadoc what's reliable when this
returns true, but keep it skinny in production. The key question is are there
any cases
[jira] [Created] (SOLR-13544) Consider adding isCloud() method as shortcut for cloudDescriptor == null checks
Gus Heck created SOLR-13544:
---
Summary: Consider adding isCloud() method as shortcut for
cloudDescriptor == null checks
Key: SOLR-13544
URL: https://issues.apache.org/jira/browse/SOLR-13544
Project: Solr
Issue Type: Improvement
Security Level: Public (Default Security Level. Issues are Public)
Components: SolrCloud
Reporter: Gus Heck
In writing SOLR-13420 I was not supper happy with the verbosity of some of my
changes, and I think a convenience method could improve code legibility.
After doing some searching I found that in the first 50ish places where
getCloudDescriptor() is called. 8 of them are followed by a null check, which
is generally necessitated in cases where the code might run in non-cloud
environments. There are a great many places where null is not checked and cloud
is assumed, presumably for reasons related to the nature of the code (i.e.
cdcr, code in zkcontroller), but some code does check, and the ethos generally
seems to be "do x if we're in cloud".
For example in MDCLoggingContext.java I see:
{code:java}
// If this is a replica of type PULL, don't create the update log
boolean skipUpdateLog = core.getCoreDescriptor().getCloudDescriptor() !=
null && !core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
This would be more concise and readable as:
{code:java}
boolean skipUpdateLog = core.isCloud() &&
!core.getCoreDescriptor().getCloudDescriptor().requiresTransactionLog();
{code}
Or in the routed alias code:
{code:java}
SolrCore core = req.getCore();
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
if (cloudDescriptor != null) {
String collectionName = cloudDescriptor.getCollectionName();
CoreContainer coreContainer = core.getCoreContainer();
ZkController zkController = coreContainer.getZkController();
... etc ...{code}
vs this which seems to read much smoother...
{code:java}
SolrCore core = req.getCore();
if (core.isCloud()) {
CoreDescriptor coreDescriptor = core.getCoreDescriptor();
CloudDescriptor cloudDescriptor = coreDescriptor.getCloudDescriptor();
String collectionName = cloudDescriptor.getCollectionName();
... etc ...
{code}
The idea would be to add to SolrCore this method
{code:java}
public boolean isCloud() {
return getCoreDescriptor().getCloudDescriptor() != null;
} {code}
We also might add some asserts in there for other things folks might want to
assume to be true in the cloud case and javadoc what's reliable when this
returns true, but keep it skinny in production. The key question is are there
any cases where we're in solr cloud but getCloudDescriptor() is null? Seems
hard to imagine, but what do others think?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13420) Allow Routed Aliases to use Collection Properties instead of core properties
[
https://issues.apache.org/jira/browse/SOLR-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16862720#comment-16862720
]
Gus Heck commented on SOLR-13420:
-
Patch with minor update to utilize the caching in SOLR-13420 plus some unit
test verification and a CHANGES.txt entry. Will commit to branch_8x and master
tomorrow evening if no objections.
> Allow Routed Aliases to use Collection Properties instead of core properties
>
>
> Key: SOLR-13420
> URL: https://issues.apache.org/jira/browse/SOLR-13420
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13420.patch, SOLR-13420.patch, SOLR-13420.patch,
> SOLR-13420.patch, SOLR-13420.patch
>
>
> The current routed alias code is relying on a core property named
> routedAliasName to detect when the Routed Alias wrapper URP should be applied
> to Distributed Update Request Processor.
> {code:java}
> #Written by CorePropertiesLocator
> #Sun Mar 03 06:21:14 UTC 2019
> routedAliasName=testalias21
> numShards=2
> collection.configName=_default
> ... etc...
> {code}
> Core properties are not changeable after the core is created, and they are
> written to the file system for every core. To support a unit test for
> SOLR-13419 I need to create some legacy formatted collection names, and
> arrange them into a TRA, but this is impossible due to the fact that I can't
> change the core property from the test. There's a TODO dating back to the
> original TRA implementation in the routed alias code to switch to collection
> properties instead, so this ticket will address that TODO to support the test
> required for SOLR-13419.
> Back compatibility with legacy core based TRA's and CRA's will of course be
> maintained. I also expect that this will facilitate some more nimble handling
> or routed aliases with future auto-scaling capabilities such as possibly
> detaching and archiving collections to cheaper, slower machines rather than
> deleting them. (presently such a collection would still attempt to use the
> routed alias if it received an update even if it were no longer in the list
> of collections for the alias)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13420) Allow Routed Aliases to use Collection Properties instead of core properties
[
https://issues.apache.org/jira/browse/SOLR-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-13420:
Attachment: SOLR-13420.patch
> Allow Routed Aliases to use Collection Properties instead of core properties
>
>
> Key: SOLR-13420
> URL: https://issues.apache.org/jira/browse/SOLR-13420
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13420.patch, SOLR-13420.patch, SOLR-13420.patch,
> SOLR-13420.patch, SOLR-13420.patch
>
>
> The current routed alias code is relying on a core property named
> routedAliasName to detect when the Routed Alias wrapper URP should be applied
> to Distributed Update Request Processor.
> {code:java}
> #Written by CorePropertiesLocator
> #Sun Mar 03 06:21:14 UTC 2019
> routedAliasName=testalias21
> numShards=2
> collection.configName=_default
> ... etc...
> {code}
> Core properties are not changeable after the core is created, and they are
> written to the file system for every core. To support a unit test for
> SOLR-13419 I need to create some legacy formatted collection names, and
> arrange them into a TRA, but this is impossible due to the fact that I can't
> change the core property from the test. There's a TODO dating back to the
> original TRA implementation in the routed alias code to switch to collection
> properties instead, so this ticket will address that TODO to support the test
> required for SOLR-13419.
> Back compatibility with legacy core based TRA's and CRA's will of course be
> maintained. I also expect that this will facilitate some more nimble handling
> or routed aliases with future auto-scaling capabilities such as possibly
> detaching and archiving collections to cheaper, slower machines rather than
> deleting them. (presently such a collection would still attempt to use the
> routed alias if it received an update even if it were no longer in the list
> of collections for the alias)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13420) Allow Routed Aliases to use Collection Properties instead of core properties
[
https://issues.apache.org/jira/browse/SOLR-13420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gus Heck updated SOLR-13420:
Description:
The current routed alias code is relying on a core property named
routedAliasName to detect when the Routed Alias wrapper URP should be applied
to Distributed Update Request Processor.
{code:java}
#Written by CorePropertiesLocator
#Sun Mar 03 06:21:14 UTC 2019
routedAliasName=testalias21
numShards=2
collection.configName=_default
... etc...
{code}
Core properties are not changeable after the core is created, and they are
written to the file system for every core. To support a unit test for
SOLR-13419 I need to create some legacy formatted collection names, and arrange
them into a TRA, but this is impossible due to the fact that I can't change the
core property from the test. There's a TODO dating back to the original TRA
implementation in the routed alias code to switch to collection properties
instead, so this ticket will address that TODO to support the test required for
SOLR-13419.
Back compatibility with legacy core based TRA's and CRA's will of course be
maintained. I also expect that this will facilitate some more nimble handling
or routed aliases with future auto-scaling capabilities such as possibly
detaching and archiving collections to cheaper, slower machines rather than
deleting them. (presently such a collection would still attempt to use the
routed alias if it received an update even if it were no longer in the list of
collections for the alias)
was:
The current routed alias code is relying on a core property named
routedAliasName to detect when the Routed Alias wrapper URP should be applied
to Distributed Update Request Processor.
{code:java}
#Written by CorePropertiesLocator
#Sun Mar 03 06:21:14 UTC 2019
routedAliasName=testalias21
numShards=2
collection.configName=_default
... etc...
{code}
Core properties are not changeable after the core is created, and they are
written to the file system for every core. To support a unit test for
SOLR-13419 I need to create some legacy formatted collection names, and arrange
them into a TRA, but this is impossible due to the fact that I can't change the
core property from the test. There's a TODO dating back to the original TRA
implementation in the routed alias code to switch to collection properties
instead, so this ticket will address that TOD to support the test required for
SOLR-13419.
Back compatibility with legacy core based TRA's and CRA's will of course be
maintained. I also expect that this will facilitate some more nimble handling
or routed aliases with future auto-scaling capabilities such as possibly
detaching and archiving collections to cheaper, slower machines rather than
deleting them. (presently such a collection would still attempt to use the
routed alias if it received an update even if it were no longer in the list of
collections for the alias)
> Allow Routed Aliases to use Collection Properties instead of core properties
>
>
> Key: SOLR-13420
> URL: https://issues.apache.org/jira/browse/SOLR-13420
> Project: Solr
> Issue Type: Sub-task
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13420.patch, SOLR-13420.patch, SOLR-13420.patch,
> SOLR-13420.patch
>
>
> The current routed alias code is relying on a core property named
> routedAliasName to detect when the Routed Alias wrapper URP should be applied
> to Distributed Update Request Processor.
> {code:java}
> #Written by CorePropertiesLocator
> #Sun Mar 03 06:21:14 UTC 2019
> routedAliasName=testalias21
> numShards=2
> collection.configName=_default
> ... etc...
> {code}
> Core properties are not changeable after the core is created, and they are
> written to the file system for every core. To support a unit test for
> SOLR-13419 I need to create some legacy formatted collection names, and
> arrange them into a TRA, but this is impossible due to the fact that I can't
> change the core property from the test. There's a TODO dating back to the
> original TRA implementation in the routed alias code to switch to collection
> properties instead, so this ticket will address that TODO to support the test
> required for SOLR-13419.
> Back compatibility with legacy core based TRA's and CRA's will of course be
> maintained. I also expect that this will facilitate some more nimble handling
> or routed aliases with future auto-scaling capabilities such as possibly
> detaching and archiving collections to cheaper, slower machines rather than
> deleting them. (presently such a collection would still attempt to use the
> routed alias if it received an update even if it were no
[jira] [Commented] (SOLR-13534) Dynamic loading of jars from a url
[
https://issues.apache.org/jira/browse/SOLR-13534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861492#comment-16861492
]
Gus Heck commented on SOLR-13534:
-
If I understand the doc edits I see in the PR, it looks like this is turned on
by {{-Denable.runtime.lib=true}} ? I'm wondering if it shouldn't have its own
flag. To me this is another level of risk beyond loading from the blob store.
With this, they only have to trick/hack you once into running the config
command... previously they had to trick you twice (once to upload to the
blobstore and once to run the config command). Thinking XSRF/person
hacking/misconfigured authorization/ etc. Also the attacker leaves no footprint
to show what it was they did.
> Dynamic loading of jars from a url
> --
>
> Key: SOLR-13534
> URL: https://issues.apache.org/jira/browse/SOLR-13534
> Project: Solr
> Issue Type: Improvement
>Reporter: Noble Paul
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Dynamic loading is possible from {{.system}} collection. It's much easier to
> host the jars on a remote service and load it from there. This way the user
> should have no problem in loading jars when the {{.system}} collection is not
> available for some reason.
> The steps should look as follows
> # get the hash of your jar file. {{openssl dgst -sha512 }}
> # upload it your hosting service . say the location is
> {{[http://host:port/my-jar/location|http://hostport/]}}
> # create a runtime lib entry for the collection as follows
> {code:java}
> curl http://localhost:8983/solr/techproducts/config -H
> 'Content-type:application/json' -d '{
>"add-runtimelib": { "name":"jarblobname",
> "sha512":"e94bb3990b39aacdabaa3eef7ca6102d96fa46766048da50269f25fd41164440a4e024d7a7fb0d5ec328cd8322bb65f5ba7886e076a8f224f78cb310fd45896d"
> , "url" : "http://host:port/my-jar/loaction"}
> }'
> {code}
> to update the jar, just repeat the steps and use the {{update-runtimelib}} to
> update the sha512 hash
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861095#comment-16861095 ] Gus Heck edited comment on SOLR-13439 at 6/11/19 2:29 PM: -- Given that this has been well discussed and the existence of the api has basically been agreed upon, I'm moving this forward so I can build on it, and if we need to tweak something about the details of the implementation we can still do that. Will be pushing to 8x shortly. was (Author: gus_heck): Given that this has been well discussed and the existence of the api has basically been agreed upon, I'm moving this forward so I can build on it, and if we need to tweak something about the details of the implementation we can still do that. > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16861095#comment-16861095 ] Gus Heck commented on SOLR-13439: - Given that this has been well discussed and the existence of the api has basically been agreed upon, I'm moving this forward so I can build on it, and if we need to tweak something about the details of the implementation we can still do that. > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13532) Unable to start core recovery due to timeout in ping request
[
https://issues.apache.org/jira/browse/SOLR-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859330#comment-16859330
]
Gus Heck commented on SOLR-13532:
-
Hard coded timeouts are a general problem in several areas. I opened SOLR-13457
a little while ago to track these sorts of problems.
> Unable to start core recovery due to timeout in ping request
>
>
> Key: SOLR-13532
> URL: https://issues.apache.org/jira/browse/SOLR-13532
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
>Affects Versions: 7.6
>Reporter: Suril Shah
>Priority: Major
>
> Discovered following issue with the core recovery:
> * Core recovery is not being initialized and throwing following exception
> message :
> {code:java}
> 2019-06-07 00:53:12.436 INFO
> (recoveryExecutor-4-thread-1-processing-n::8983_solr
> x:_shard41_replica_n2777 c: s:shard41
> r:core_node2778) x:_shard41_replica_n2777
> o.a.s.c.RecoveryStrategy Failed to connect leader http://:8983/solr
> on recovery, try again{code}
> * Above error occurs when ping request takes time more than a timeout period
> which is hard-coded to one second in solr source code. However In a general
> production setting it is common to have ping time more than one second,
> hence, the core recovery never starts and exception is thrown.
> * Also the other major concern is that this exception is logged as an info
> message, hence it is very difficult to identify the error if info logging is
> not enabled.
> * Please refer to following code snippet from the [source
> code|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L789-L803]
> to understand the above issue.
> {code:java}
> try (HttpSolrClient httpSolrClient = new
> HttpSolrClient.Builder(leaderReplica.getCoreUrl())
> .withSocketTimeout(1000)
> .withConnectionTimeout(1000)
>
> .withHttpClient(cc.getUpdateShardHandler().getRecoveryOnlyHttpClient())
> .build()) {
> SolrPingResponse resp = httpSolrClient.ping();
> return leaderReplica;
> } catch (IOException e) {
> log.info("Failed to connect leader {} on recovery, try again",
> leaderReplica.getBaseUrl());
> Thread.sleep(500);
> } catch (Exception e) {
> if (e.getCause() instanceof IOException) {
> log.info("Failed to connect leader {} on recovery, try again",
> leaderReplica.getBaseUrl());
> Thread.sleep(500);
> } else {
> return leaderReplica;
> }
> }
> {code}
> The above issue will have high impact in production level clusters, since
> cores not being able to recover may lead to data loss.
> Following improvements would be really helpful:
> 1. The [timeout for ping
> request|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L790-L791]
> in *RecoveryStrategy.java* should be configurable and the defaults set to
> high values like 15seconds.
> 2. The exception message in [line
> 797|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L797]
> and [line
> 801|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/cloud/RecoveryStrategy.java#L801]
> in *RecoveryStrategy.java* should be logged as *error* messages instead of
> *info* messages
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13516) support of apache solr for ub 19.04
[ https://issues.apache.org/jira/browse/SOLR-13516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13516. - Resolution: Invalid Apache's Jira for Lucene/Solr is not a support portal, it's for development only. Please use the mailing list for this type of question. Jira is for when: * You have (or will soon have) a patch or pull request with direct code changes that need review * You have steps to reproduce, (from a fresh install if possible). * You have proposed and discussed a new feature or bug fix on the mailing list, got a positive reception and want to move to contributing that feature. > support of apache solr for ub 19.04 > --- > > Key: SOLR-13516 > URL: https://issues.apache.org/jira/browse/SOLR-13516 > Project: Solr > Issue Type: Test > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: 8.0 >Reporter: ketan kunde >Priority: Major > > I have been trying to build apache solr 8.1.1 for ub 19.04 > Build is successful but the solr server does not start as expected. > I want to know whether apache solr is been supported on ub 19.04 or not. > > Regards > Ketan Kunde -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13508) Solr.log file was not updating with latest information
[ https://issues.apache.org/jira/browse/SOLR-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13508: Priority: Major (was: Blocker) > Solr.log file was not updating with latest information > -- > > Key: SOLR-13508 > URL: https://issues.apache.org/jira/browse/SOLR-13508 > Project: Solr > Issue Type: Wish > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: raja >Priority: Major > Labels: solr.log > > Since last 4 months solr.log file was not updating/growing with the latest > solr logs information. we have properly configured the log4j.properties file > and log4j.xml files. whenever we are restarting the solr service the only > "solr-8983-console.log" updated with solr restart time stamp. we have > verified the solr.log file location on server level with find command but > it's showing only what we configured on .properties file location. > Need your help on how solr admin console showing latest logs information and > From where it is pulling the latest logs from server. > Solr version :7.5 Server type: linux -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13508) Solr.log file was not updating with latest information
[ https://issues.apache.org/jira/browse/SOLR-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13508. - Resolution: Invalid > Solr.log file was not updating with latest information > -- > > Key: SOLR-13508 > URL: https://issues.apache.org/jira/browse/SOLR-13508 > Project: Solr > Issue Type: Wish > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: raja >Priority: Blocker > Labels: solr.log > > Since last 4 months solr.log file was not updating/growing with the latest > solr logs information. we have properly configured the log4j.properties file > and log4j.xml files. whenever we are restarting the solr service the only > "solr-8983-console.log" updated with solr restart time stamp. we have > verified the solr.log file location on server level with find command but > it's showing only what we configured on .properties file location. > Need your help on how solr admin console showing latest logs information and > From where it is pulling the latest logs from server. > Solr version :7.5 Server type: linux -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13508) Solr.log file was not updating with latest information
[ https://issues.apache.org/jira/browse/SOLR-13508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16854592#comment-16854592 ] Gus Heck commented on SOLR-13508: - Apache's Jira is not a support portal, it's for development only. Any time you feel like writing the word "help" this is the wrong place to write it. Please use the mailing list. Jira is for when: # You have (or will soon have) a patch or pull request with direct code changes that need review # You have steps to reproduce, (from a fresh install if possible). # You have proposed and discussed a new feature or bug fix on the list, got a positive reception and want to move to contributing that feature. > Solr.log file was not updating with latest information > -- > > Key: SOLR-13508 > URL: https://issues.apache.org/jira/browse/SOLR-13508 > Project: Solr > Issue Type: Wish > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Reporter: raja >Priority: Blocker > Labels: solr.log > > Since last 4 months solr.log file was not updating/growing with the latest > solr logs information. we have properly configured the log4j.properties file > and log4j.xml files. whenever we are restarting the solr service the only > "solr-8983-console.log" updated with solr restart time stamp. we have > verified the solr.log file location on server level with find command but > it's showing only what we configured on .properties file location. > Need your help on how solr admin console showing latest logs information and > From where it is pulling the latest logs from server. > Solr version :7.5 Server type: linux -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16852003#comment-16852003 ] Gus Heck commented on SOLR-13439: - Patch using expiration on the PropsWatcher instead of an expiring cache. In addition, any get without a cache duration takes the opportunity to evict the cached value if it should be expired anyway. I also decided to timestamp the cached properties and actively clean them up as well (only if no CollectionPropsWatcher is active of course). [~tomasflobbe] If this version seems suitable to you I'd like to commit this soon. > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13439: Attachment: SOLR-13439.patch > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13439: Attachment: (was: SOLR-13439) > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13439: Attachment: SOLR-13439 > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851921#comment-16851921 ] Gus Heck commented on SOLR-13491: - Got lucky and found a rare failure while committing to 8x so touched up the the test. > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Priority: Major > Attachments: SOLR-13491.patch, SOLR-13491.patch > > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Resolved] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck resolved SOLR-13491. - Resolution: Fixed Assignee: Gus Heck Fix Version/s: 8.2 master (9.0) > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Fix For: master (9.0), 8.2 > > Attachments: SOLR-13491.patch, SOLR-13491.patch > > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16851525#comment-16851525 ] Gus Heck commented on SOLR-13491: - patch with unit test, will commit tomorrow assuming tests/precommit happy. > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Priority: Major > Attachments: SOLR-13491.patch, SOLR-13491.patch > > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13491: Attachment: SOLR-13491.patch > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Priority: Major > Attachments: SOLR-13491.patch, SOLR-13491.patch > > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850988#comment-16850988 ] Gus Heck commented on SOLR-13439: - I realized two things: First, my changes for SOLR-13418 have made it possible to easily distinguish deletion from removing the last property on a collection that still exists (via a value of -1 for zk version that I chose to return in this case and the existence of a prior ZK version other than -1 for our cached properties). That allows us to solves all the sticky problems about how to get rid of stuff when collections have been deleted. Second I did misunderstand your proposal. When you said _optionally set a “timed watch”_, I thought you meant a CollectionPropertiesWatcher similar to your prior suggestions of "just setting a watch". I still had your old suggestions in my head, but now I realize you only meant altering PropsWatcher, which is much better. Taken together, I think those realizations mean that your way works to my satisfaction and is cleaner too... Most of my reason for the expiring caches was to guard against orphaned watches and things that might accumulate, but we now have good control of what happens when a collection disappears so the "quiet" case where watches never fire is our worst case and that's not soluble by either solution, but can be limited to just one watch per collection once SOLR-13491 is in. Also the above discussion has helped clarify ZK's actual behavior (to me at least :) ) and several things I worried about initially are non-issues. > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13497) Download page must not redirect to mirror page
[ https://issues.apache.org/jira/browse/SOLR-13497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850963#comment-16850963 ] Gus Heck commented on SOLR-13497: - Should https be used by default? > Download page must not redirect to mirror page > -- > > Key: SOLR-13497 > URL: https://issues.apache.org/jira/browse/SOLR-13497 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Sebb >Assignee: Jan Høydahl >Priority: Major > > The direct download page [1] currently redirects to the mirror system [2] > after a few seconds. > I think that is wrong; users should be encouraged to verify the downloads, > but the redirect makes verification harder. > The normal download page [3] is OK. > [1] [http://lucene.apache.org/solr/mirrors-solr-latest-redir.html] > [2] http://www.apache.org/dyn/closer.lua/lucene/solr/8.1.1 > [3] http://lucene.apache.org/solr/downloads.html -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13490) waitForState/registerCollectionStateWatcher can see stale liveNodes data due to (Zk) Watcher race condition
[
https://issues.apache.org/jira/browse/SOLR-13490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850933#comment-16850933
]
Gus Heck commented on SOLR-13490:
-
I tend to like #3 because it simplifies the API such that it's only watching
one thing at a time which is simpler. Your hypothetical watcher sounds like it
wants to watch two things (live nodes and state) and I like the solution to not
conflate the two items. The watcher can watch one or the other and query when
it sees a change, or it can watch both and keep a state machine if it prefers.
The watching code will know which strategy is best/most efficient for whatever
it does (or get it wrong, but not require changes to zkStateReader when that's
discovered). Passing back the ZkStateReader is likely redundant for code that
has set a watch (via a reference zkStateReader it already holds)
> waitForState/registerCollectionStateWatcher can see stale liveNodes data due
> to (Zk) Watcher race condition
> ---
>
> Key: SOLR-13490
> URL: https://issues.apache.org/jira/browse/SOLR-13490
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Hoss Man
>Priority: Major
> Attachments: SOLR-13490.patch
>
>
> I was investigating some failures in
> {{TestCloudSearcherWarming.testRepFactor1LeaderStartup}} which lead me to the
> hunch that {{waitForState}} wasn't ensuring that the predicates registered
> would always be called if/when a node was shutdown.
> Digging into it a bit more, I found that the root cause seems to be the way
> the {{CollectionStateWatcher}} / {{CollectionStatePredicate}} APIs pass in
> *both* the {{DocCollection}}, and the "current" {{liveNodes}} - but are only
> _triggered_ by the {{StateWatcher}} on the {{state.json}} (which is used to
> rebuild the {{DocCollection}}) - when the {{CollectionStateWatcher}} /
> {{CollectionStatePredicate}} are called, they get the "fresh"
> {{DocCollection}} but they get the _cached_ {{ZkStateReader.liveNodes}}
> Meanwhile, the {{LiveNodeWatcher}} only calls {{refreshLiveNodes()}} only
> updates {{ZkStateReader.liveNodes}} and triggers any {{LiveNodesListener}} -
> it does *NOT* invoke any {{CollectionStateWatcher}} that may have replicas
> hosted on any of changed nodes.
> Since there is no garunteed order that Watchers will be triggered, this means
> there is a race condition where the following can happen...
> * client1 has a ZkStateReader with cached {{liveNodes=[N1, N2, N3]}}
> * client1 registers a {{CollectionStateWatcher}} "watcherZ" that cares if
> "replicaX" of collectionA is on a "down" node
> * client2 causes shutdown of node N1 which is hosting replicaX
> * client1's zkStateReader gets a WatchedEvent for state.json of collectionA
> ** DocCollection for collectionA is rebuilt
> ** watcherZ is fired w/cached {{liveNodes=[N1, N2, N3]}} and the new
> DocCollection
> *** watcherZ sees that replicaX is on N1, but thinks N1 is live
> *** watcherZ says "everything ok, not the event i was waiting for" and
> doesn't take any action
> * client1's zkStateReader gets a WatchedEvent for LIVE_NODES_ZKNODE
> ** zkStateReader.liveNodes is rebuilt
> ...at no point in this sequence (or after this) will watcherZ be notified
> fired with the updated liveNodes (unless/until another {{state.json}} change
> is made for collectionA.
>
> While this is definitely be problematic in _tests_ that deal with node
> lifecyle and use things like {{SolrCloudTestCase.waitForState(...,
> SolrCloudTestCase.clusterShape(...))}} to check for the expected
> shards/replicas, a cursory search of how/where
> {{ZkStateReader.waitForState(...)}} and
> {{ZkStateReader.registerCollectionStateWatcher(...)}} are used in solr-core
> suggests that could also lead to bad behavior in situations like reacting to
> shard leader loss, waiting for all leaders of SYSTEM_COLL to come online for
> upgrade, running PrepRecoveryOp, etc... (anywhere that liveNodes is used by
> the watcher/predicate)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[ https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848995#comment-16848995 ] Gus Heck commented on SOLR-13439: - Bullet #3 of my counter proposal has been moved to SOLR-13491 since it is of value regardless of this ticket and so that it can be more easily tracked/discussed in it's own right. > Make collection properties easier and safer to use in code > -- > > Key: SOLR-13439 > URL: https://issues.apache.org/jira/browse/SOLR-13439 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: master (9.0) >Reporter: Gus Heck >Assignee: Gus Heck >Priority: Major > Attachments: SOLR-13439.patch, SOLR-13439.patch > > > (breaking this out from SOLR-13420, please read there for further background) > Before this patch the api is quite confusing (IMHO): > # any code that wanted to know what the properties for a collection are > could call zkStateReader.getCollectionProperties(collection) but this was a > dangerous and trappy API because that was a query to zookeeper every time. If > a naive user auto-completed that in their IDE without investigating, heavy > use of zookeeper would ensue. > # To "do it right" for any code that might get called on a per-doc or per > request basis one had to cause caching by registering a watcher. At which > point the getCollectionProperties(collection) magically becomes safe to use, > but the watcher pattern probably looks famillar induces a user who hasn't > read the solr code closely to create their own cache and update it when their > watcher is notified. If the caching side effect of watches isn't understood > this will lead to many in-memory copies of collection properties maintained > in user code. > # This also creates a task to be scheduled on a thread (PropsNotification) > and induces an extra thread-scheduling lag before the changes can be observed > by user code. > # The code that cares about collection properties needs to have a lifecycle > tied to either a collection or something other object with an even more > ephemeral life cycle such as an URP. The user now also has to remember to > ensure the watch is unregistered, or there is a leak. > After this patch > # Calls to getCollectionProperties(collection) are always safe to use in any > code anywhere. Caching and cleanup are automatic. > # Code that really actually wants to know if a collection property changes > so it can wake up and do something (autoscaling?) still has the option of > registering a watcher that will asynchronously send them a notification. > # Updates can be observed sooner via getCollectionProperties with no need to > wait for a thread to run. (vs a cache held in user code) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13491: Attachment: SOLR-13491.patch > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Priority: Major > Attachments: SOLR-13491.patch > > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13491) SolrZkClient can register the same watch twice.
[ https://issues.apache.org/jira/browse/SOLR-13491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16848255#comment-16848255 ] Gus Heck commented on SOLR-13491: - Not at my desk, but the easy check is to see if watch registration in zookeeper 3.5.5 still lands in a map of sets (there's a map for data watches, one for exists watches and one other I don't recall atm.) The goal of this is to let those sets do their thing. I don't recall seeing changes to wrapWatch() in your patch, which is all that would change on the solr side. > SolrZkClient can register the same watch twice. > --- > > Key: SOLR-13491 > URL: https://issues.apache.org/jira/browse/SOLR-13491 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) > Components: SolrCloud >Affects Versions: 8.0 >Reporter: Gus Heck >Priority: Major > > While working on SOLR-13420 and related tickets I noticed that we presently > use the wrapWatch() method to wrap all zookeeper watches. This allows the > watch to be processed asynchronously by an executor which is good, but it > subtly changes the way zookeeper behaves. Normally zookeeper keeps a > Map> that ensures that if the same watch is registered > twice it's only notified once. However, our wrapper breaks this because now, > if the same watch is added twice each one gets it's own wrapper with default > Object.equals() and thus the two wrapped versions of the Watch aren not > .equals() and may co-exist in the Set > I do not know of any current cases for which this would cause a problem, but > changes I am contemplating for SOLR-13375 are likely to hit this case and at > a minimum create wasted re-notifications of the same event and excess watch > load in Zk. > Moving the anonymous class wrapping to an explicit inner class and writing > hashcode/equals carefully to act as a pass through such that any to wrappers > with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Created] (SOLR-13491) SolrZkClient can register the same watch twice.
Gus Heck created SOLR-13491: --- Summary: SolrZkClient can register the same watch twice. Key: SOLR-13491 URL: https://issues.apache.org/jira/browse/SOLR-13491 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Components: SolrCloud Affects Versions: 8.0 Reporter: Gus Heck While working on SOLR-13420 and related tickets I noticed that we presently use the wrapWatch() method to wrap all zookeeper watches. This allows the watch to be processed asynchronously by an executor which is good, but it subtly changes the way zookeeper behaves. Normally zookeeper keeps a Map> that ensures that if the same watch is registered twice it's only notified once. However, our wrapper breaks this because now, if the same watch is added twice each one gets it's own wrapper with default Object.equals() and thus the two wrapped versions of the Watch aren not .equals() and may co-exist in the Set I do not know of any current cases for which this would cause a problem, but changes I am contemplating for SOLR-13375 are likely to hit this case and at a minimum create wasted re-notifications of the same event and excess watch load in Zk. Moving the anonymous class wrapping to an explicit inner class and writing hashcode/equals carefully to act as a pass through such that any to wrappers with the same watch object are .equals() should fix this. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[
https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16845003#comment-16845003
]
Gus Heck commented on SOLR-13439:
-
The timed watches suggestion has the following attributes that make me
nervous...
* Each watch implies an invocation of onStateChanged by the thread spawned for
each notification which is something of a waste, though that notification would
be a trivial {{return false}}, that still gets evaluated by the for loop over
all watches.
* From a usability and api design standpoint I really don't like setting a
watch you plan to ignore just for the caching side effect of the watch's
existence. It's just not straightforward. If you want to get notified when
something happens, set a watch, if you want caching the api should be clear
that that's what you're requesting (clarity improvements below).
* It's unclear to me how I avoid adding a watch on every request if I want to
check a collection property during a request. I don't know a way of
distinguishing my watch from a watch set by someone else (who might care about
the notification via onStateChanged), and if I add/remove one every request
it's not clear I'm getting caching between requests (even if it did somehow
retain the caching, that's very counter intuitive for users of the API?)
* Even if I simply trust in the side effects due to the existence of someone
else's watch it then becomes possible (via that watch returning true from
onStateChanged) for the cache to disappear out from under me and suddenly calls
to getCollectionProperties() are all expensive again.
How about this counter proposal:
* The existing getCollectionProperties() method remain a simple single shot
call that will pull from the cache or get from zk as it does now and not effect
caching in any way.
* My logic for expiration based caching be moved to a new method
getAndCacheCollectionProperties()
* The anonymous wrapper in {{wrapWatcher(final Watcher watcher)}} be converted
to a real class that performs a pass through in equals/hashcode for the
underlying watcher instance, thus allowing code that sends in the same watch
object to have the zkClient function as intended and not accumulate duplicates
that differ only by their anonymous wrapper.
* Retain a set of PropsWatcher objects such that we can re-use them and thus
avoid creating duplicate watches. These objects are very lightweight, but if we
want to worry about long running systems accumulating these objects as
collections come and go from the node it can also be an expiring cache with a
much longer time scale (days), and we accept the duplication of watches on a
time scale of days as inconsequential.
* When SOLR-8346 becomes available, add a cache expiration callback to
leverage ZOOKEEPER-442.
The fact that caching and watches for properties are tied to each other in a
way that is visible to callers of the API seems unnecessarily complicated, but
probably can't be fixed for backward compatibility reasons. If I could start
from scratch, watching would only provide notification of changes and have no
visible side effects, and caching would be an optimization option available for
fetching, not related to watches. Under the covers they might cooperate to
avoid multiple watches, but that should be undetectable to the user of the API.
> Make collection properties easier and safer to use in code
> --
>
> Key: SOLR-13439
> URL: https://issues.apache.org/jira/browse/SOLR-13439
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
> Components: SolrCloud
>Affects Versions: master (9.0)
>Reporter: Gus Heck
>Assignee: Gus Heck
>Priority: Major
> Attachments: SOLR-13439.patch, SOLR-13439.patch
>
>
> (breaking this out from SOLR-13420, please read there for further background)
> Before this patch the api is quite confusing (IMHO):
> # any code that wanted to know what the properties for a collection are
> could call zkStateReader.getCollectionProperties(collection) but this was a
> dangerous and trappy API because that was a query to zookeeper every time. If
> a naive user auto-completed that in their IDE without investigating, heavy
> use of zookeeper would ensue.
> # To "do it right" for any code that might get called on a per-doc or per
> request basis one had to cause caching by registering a watcher. At which
> point the getCollectionProperties(collection) magically becomes safe to use,
> but the watcher pattern probably looks famillar induces a user who hasn't
> read the solr code closely to create their own cache and update it when their
> watcher is notified. If the caching side effect of watches isn't understood
> this will lead
[jira] [Commented] (SOLR-13439) Make collection properties easier and safer to use in code
[
https://issues.apache.org/jira/browse/SOLR-13439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840611#comment-16840611
]
Gus Heck commented on SOLR-13439:
-
{quote}Exactly, which is why the option of not setting a watch exists for cases
that only need infrequent access.
{quote}
If that option is truly important, we can have 2 methods, one that does cause a
watch/cache and one that hits zookeeper every time, getCollectionProperties()
and getCachedCollectionProperties() perhaps?
{quote}Not on only those cases. There are two more leak scenarios that I’d like
to consider:
# Even if the PropsWatcher expires and it’s removed from the cache, ZooKeeper
server will still maintain the watch referenced to this zkClient, that means we
just keep accumulating them for every call to getCollectionProperties. If you
do this to many collections and collection properties changes are infrequent,
this may become a problem. I believe the PropsWatcher object won’t be garbage
collected, since it includes the callback.{quote}
ZkStateReader lives at the CoreContainer level. so we have one per node, and
thus one instance of the zk client. Looking into the zookeeper code, what I
think I see is a map of path (String) to HashSet, but the watchers
added to that set are instances of ServerCnxn.java (which implements watcher)
which appears to also be the object representing the connection to the client,
of which (I think) there will only be one per client. So the max load on the
zookeeper server side for a given path is set by the number of Solr nodes
observing that path. So the absolute worst case for the server in this code is
{{collections X nodes}} if every collection has a shard on every node, the
properties never change and every collection has it's properties queried. The
worst case hasn't changed, though it does become more likely with the direction
I'm heading.
According to ZOOKEEPER-1177 a watch is about 100 bytes... so a 1000 collections
on 100 nodes (100k watches) could generate up to 10Mb of watches on the server.
According to that issue, performance of the "old" (which is what 3.4.14 would
be since that issue is fixed for 3.6) is:
{code:java}
OLD
[junit] 9736ms to add
[junit] size:1000
[junit] 0ms to size
[junit] 5615ms to dumpwatches true
[junit] 3035ms to dumpwatches false
[junit] 5530ms to trigger{code}
those numbers are for 10M watches so those operations on our 100K example is
likely to run in the tens of ms. (the foregoing assuming I understood the code
and the issue correctly, and a linear relationship vs number of watches).
In zookeeper 3.5 we'll also have the opportunity to cull out watches on with
ZOOKEEPER-442
So server side in zk, it looks like we should be able to handle some fairly
beefy use cases, what's the max nodes times collections you think we need to
support? Do you have large test clusters on which to validate these effects? (I
do not unfortunately).
As for GC I did think about that... references to a PropsWatcher are held in 2
places:
# The ConditionalExpiringCache as a value
# The closure created when the anonymous implementation of SolrZkWatcher is
created
Once the watch fires, the anonymous wrapper should not be referenced by zk code
and become held by a lambda submitted to the executor, which in turn is
discarded once the call to process() finishes. Once the watch has fired and the
cache has expired there are no references held and it becomes GC eligible.
GC is prevented however if the result of process() (called by the lambda) is to
call refreshAndWatch() (in the event that a CollectionPropsWatch still exists)
since that will put it back in the ConditionalExpiringCache with a renewed
timestamp and re-register the watch for zk.
{quote}2 Even if you keep calling getCollectionProperties on the same
collection, but after expiration, you keep adding new watches, for the same
path. That not only means leaks of watches in ZooKeeper server and PropsWatcher
objects in Solr, but also that, if at some point later in time a collection
property changes all those watches are going to fire at the same time. Even
more, if at the time of the fire, the collection is in the cache (because of a
recent call to getCollectionProperties), all those watches for the same path,
will re-fetch the collections properties file and then re-set. This arguably
can happen today, if someone uses watchers in the same way (add/remove watches
rapidly), but the idea of the watch is to be longer lived (i.e. the life of the
core), while with the patch it would happen with components that just call
getCollectionProperties with some frequency (i.e. per request, in cases with
low request rate)
{quote}
Some further research shows that we may have a subtle preexisting problem
there. Zookeeper keeps a set of watches so adding the same watch object to it
repeatedly is not a big deal at
[jira] [Commented] (SOLR-13457) Managing Timeout values in Solr
[ https://issues.apache.org/jira/browse/SOLR-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837569#comment-16837569 ] Gus Heck commented on SOLR-13457: - I'm also thinking that maybe instead of invoking start.jar directly it will be time to put a thin wrapper around that invocation so we can influence what jetty does from our own configs & set a System property to let ourselves know if we are a solr server, or if we've been run inside someone else's code, and if a server if we are in cloud or classic mode, because we'll obviously want to load stuff from zk when we're a cloud, but that option is not available in the other two cases. > Managing Timeout values in Solr > --- > > Key: SOLR-13457 > URL: https://issues.apache.org/jira/browse/SOLR-13457 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: master (9.0) >Reporter: Gus Heck >Priority: Major > > Presently, Solr has a variety of timeouts for various connections or > operations. These timeouts have been added, tweaked and refined and in some > cases made configurable in an ad-hoc manner by the contributors of individual > features. Throughout the history of the project. This is all well and good > until one experiences a timeout during an otherwise valid use case and needs > to adjust it. > This has also made managing timeouts in unit tests "interesting" as noted in > SOLR-13389. > Probably nobody has the spare time to do a tour de force through the code and > coordinate every single timeout, so in this ticket I'd like to establish a > framework for categorizing time outs, a standard for how we make each > category configurable, and then add sub-tickets to address individual > timeouts. > The intention is that eventually, there will be no "magic number" timeout > values in code, and one can predict where to find the configuration for a > timeout by determining it's category. > Initial strawman categories (feel free to knock down or suggest alternatives): > # *Feature-Instance Timeout*: Timeouts that relate to a particular > instantiation of a feature, for example a database connection timeout for a > connection to a particular database by DIH. These should be set in the > configuration of that instance. > # *Optional Feature Timeout*: A timeout that only has meaning in the context > of a particular feature that is not required for solr to function... i.e. > something that can be turned on or off. Perhaps a timeout for communication > with an external ldap for authentication purposes. These should be configured > in the same configuration that enables this feature. > # *Global System Timeout*: A timeout that will always be an active part of > Solr these should be configured in a new section of solr.xml. For > example the Jetty thread idle timeout, or the default timeout for http calls > between nodes. > # *Node Specific Timeout*: A timeout which may differ on different nodes. I > don't know of any of these, but I'll grant the possibility. These (and only > these) should be set by setting system properties. If we don't have any of > these, that's just fine :). > # *Client Timeout*: These are timeouts in solrj code that are active in code > running outside the server. They should be configurable via java api, and via > a config file of some sort from a single location defined in a sysprop or > sourced from classpath (in that order). When run on the server, the solrj > code should look for a *Global System Timeout* setting before consulting > sysprops or classpath. > *Note that in no case is a hard-coded value the correct solution.* > If we get a consensus on categories and their locations, then the next step > is to begin adding sub tickets to bring specific timeouts into compliance. > Every such ticket should include an update to the section of the ref guide > documenting the configuration to which the timeout has been added (e.g. docs > for solr.xml for Global System Timeouts) describing what exactly is affected > by the timeout, the maximum allowed value and how zero and negative numbers > are handled. > It is of course true that some of these values will have the potential to > destroy system performance or integrity, and that should be mentioned in the > update to documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13457) Managing Timeout values in Solr
[ https://issues.apache.org/jira/browse/SOLR-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837563#comment-16837563 ] Gus Heck commented on SOLR-13457: - Ah thx for the link to that issue. It occurs to me that my thinking is somewhat biased towards cloud... so there's that cloud/classic variability to account for too. > Managing Timeout values in Solr > --- > > Key: SOLR-13457 > URL: https://issues.apache.org/jira/browse/SOLR-13457 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Affects Versions: master (9.0) >Reporter: Gus Heck >Priority: Major > > Presently, Solr has a variety of timeouts for various connections or > operations. These timeouts have been added, tweaked and refined and in some > cases made configurable in an ad-hoc manner by the contributors of individual > features. Throughout the history of the project. This is all well and good > until one experiences a timeout during an otherwise valid use case and needs > to adjust it. > This has also made managing timeouts in unit tests "interesting" as noted in > SOLR-13389. > Probably nobody has the spare time to do a tour de force through the code and > coordinate every single timeout, so in this ticket I'd like to establish a > framework for categorizing time outs, a standard for how we make each > category configurable, and then add sub-tickets to address individual > timeouts. > The intention is that eventually, there will be no "magic number" timeout > values in code, and one can predict where to find the configuration for a > timeout by determining it's category. > Initial strawman categories (feel free to knock down or suggest alternatives): > # *Feature-Instance Timeout*: Timeouts that relate to a particular > instantiation of a feature, for example a database connection timeout for a > connection to a particular database by DIH. These should be set in the > configuration of that instance. > # *Optional Feature Timeout*: A timeout that only has meaning in the context > of a particular feature that is not required for solr to function... i.e. > something that can be turned on or off. Perhaps a timeout for communication > with an external ldap for authentication purposes. These should be configured > in the same configuration that enables this feature. > # *Global System Timeout*: A timeout that will always be an active part of > Solr these should be configured in a new section of solr.xml. For > example the Jetty thread idle timeout, or the default timeout for http calls > between nodes. > # *Node Specific Timeout*: A timeout which may differ on different nodes. I > don't know of any of these, but I'll grant the possibility. These (and only > these) should be set by setting system properties. If we don't have any of > these, that's just fine :). > # *Client Timeout*: These are timeouts in solrj code that are active in code > running outside the server. They should be configurable via java api, and via > a config file of some sort from a single location defined in a sysprop or > sourced from classpath (in that order). When run on the server, the solrj > code should look for a *Global System Timeout* setting before consulting > sysprops or classpath. > *Note that in no case is a hard-coded value the correct solution.* > If we get a consensus on categories and their locations, then the next step > is to begin adding sub tickets to bring specific timeouts into compliance. > Every such ticket should include an update to the section of the ref guide > documenting the configuration to which the timeout has been added (e.g. docs > for solr.xml for Global System Timeouts) describing what exactly is affected > by the timeout, the maximum allowed value and how zero and negative numbers > are handled. > It is of course true that some of these values will have the potential to > destroy system performance or integrity, and that should be mentioned in the > update to documentation. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (SOLR-13457) Managing Timeout values in Solr
[ https://issues.apache.org/jira/browse/SOLR-13457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gus Heck updated SOLR-13457: Description: Presently, Solr has a variety of timeouts for various connections or operations. These timeouts have been added, tweaked and refined and in some cases made configurable in an ad-hoc manner by the contributors of individual features. Throughout the history of the project. This is all well and good until one experiences a timeout during an otherwise valid use case and needs to adjust it. This has also made managing timeouts in unit tests "interesting" as noted in SOLR-13389. Probably nobody has the spare time to do a tour de force through the code and coordinate every single timeout, so in this ticket I'd like to establish a framework for categorizing time outs, a standard for how we make each category configurable, and then add sub-tickets to address individual timeouts. The intention is that eventually, there will be no "magic number" timeout values in code, and one can predict where to find the configuration for a timeout by determining it's category. Initial strawman categories (feel free to knock down or suggest alternatives): # *Feature-Instance Timeout*: Timeouts that relate to a particular instantiation of a feature, for example a database connection timeout for a connection to a particular database by DIH. These should be set in the configuration of that instance. # *Optional Feature Timeout*: A timeout that only has meaning in the context of a particular feature that is not required for solr to function... i.e. something that can be turned on or off. Perhaps a timeout for communication with an external ldap for authentication purposes. These should be configured in the same configuration that enables this feature. # *Global System Timeout*: A timeout that will always be an active part of Solr these should be configured in a new section of solr.xml. For example the Jetty thread idle timeout, or the default timeout for http calls between nodes. # *Node Specific Timeout*: A timeout which may differ on different nodes. I don't know of any of these, but I'll grant the possibility. These (and only these) should be set by setting system properties. If we don't have any of these, that's just fine :). # *Client Timeout*: These are timeouts in solrj code that are active in code running outside the server. They should be configurable via java api, and via a config file of some sort from a single location defined in a sysprop or sourced from classpath (in that order). When run on the server, the solrj code should look for a *Global System Timeout* setting before consulting sysprops or classpath. *Note that in no case is a hard-coded value the correct solution.* If we get a consensus on categories and their locations, then the next step is to begin adding sub tickets to bring specific timeouts into compliance. Every such ticket should include an update to the section of the ref guide documenting the configuration to which the timeout has been added (e.g. docs for solr.xml for Global System Timeouts) describing what exactly is affected by the timeout, the maximum allowed value and how zero and negative numbers are handled. It is of course true that some of these values will have the potential to destroy system performance or integrity, and that should be mentioned in the update to documentation. was: Presently, Solr has a variety of timeouts for various connections or operations. These timeouts have been added, tweaked and refined and in some cases made configurable in an ad-hoc manner by the contributors of individual features. Throughout the history of the project. This is all well and good until one experiences a timeout during an otherwise valid use case and needs to adjust it. This has also made managing timeouts in unit tests "interesting" as noted in SOLR-13389. Probably nobody has the spare time to do a tour de force through the code and coordinate every single timeout, so in this ticket I'd like to establish a framework for categorizing time outs, a standard for how we make each category configurable, and then add sub-tickets to address individual timeouts. The intention is that eventually, there will be no "magic number" timeout values in code, and one can predict where to find the configuration for a timeout by determining it's category. Initial strawman categories (feel free to knock down or suggest alternatives): # *Feature-Instance Timeout*: Timeouts that relate to a particular instantiation of a feature, for example a database connection timeout for a connection to a particular database by DIH. These should be set in the configuration of that instance. # *Optional Feature Timeout*: A timeout that only has meaning in the context of a particular feature that is not required for solr to function... i.e. something that can be
[jira] [Commented] (SOLR-13413) suspicious test failures caused by jetty TimeoutException related to using HTTP2
[ https://issues.apache.org/jira/browse/SOLR-13413?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16837273#comment-16837273 ] Gus Heck commented on SOLR-13413: - I've seen this timeout in client logs so this is a real bug in the system, existing in 8.0. > suspicious test failures caused by jetty TimeoutException related to using > HTTP2 > > > Key: SOLR-13413 > URL: https://issues.apache.org/jira/browse/SOLR-13413 > Project: Solr > Issue Type: Bug > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Hoss Man >Assignee: Cao Manh Dat >Priority: Major > Attachments: > nocommit_TestDistributedStatsComponentCardinality_trivial-no-http2.patch > > > There is evidence in some recent jenkins failures that we may have some manor > of bug in our http2 client/server code that can cause intra-node query > requests to stall / timeout non-reproducibly. > In at least one known case, forcing the jetty & SolrClients used in the test > to use http1.1, seems to prevent these test failures. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
