[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155870#comment-17155870
]
ASF subversion and git services commented on SOLR-14354:
Commit 21d811d29615ac47bc51e12a4f9f216af8463c3f in lucene-solr's branch
refs/heads/branch_8x from Cao Manh Dat
[ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=21d811d ]
SOLR-14354: Fix compile errors after cherry-pick
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
>
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] *status*: alpha *speed*: ludicrous *tests***: * *core*: passing with ignores (not solid*) * *solrj*: passing with ignores (not solid*) * *test-framework*: passing with ignores (not solid*) * *contrib/analysis-extras*: passing with ignores (not solid*) * *contrib/analytics*: passing with ignores (not solid*) * *contrib/clustering*: passing with ignores (not solid*) * *contrib/dataimporthandler*: passing with ignores (not solid*) * *contrib/dataimporthandler-extras*: passing with ignores (not solid*) * *contrib/extraction*: passing with ignores (not solid*) * *contrib/jaegertracer-configurator*: passing with ignores (not solid*) * *contrib/langid*: passing with ignores (not solid*) * *contrib/prometheus-exporter*: passing with ignores (not solid*) * *contrib/velocity*: passing with ignores (not solid*) _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ _** Non Nightly currently, Nightly comes last._ was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] *status*: alpha *speed*: ludicrous *tests***: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ _** Non Nightly currently, Nightly comes last._ > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: passing with ignores (not solid*) > * *test-framework*: passing with ignores (not solid*) > * *contrib/analysis-extras*: passing with ignores (not solid*) > * *contrib/analytics*: passing with ignores (not solid*) > * *contrib/clustering*: passing with ignores (not solid*) > * *contrib/dataimporthandler*: passing with ignores (not solid*) > * *contrib/dataimporthandler-extras*: passing with ignores (not solid*) > * *contrib/extraction*: passing with ignores (not solid*) > * *contrib/jaegertracer-configurator*: passing with ignores (not solid*) > * *contrib/langid*: passing with ignores (not solid*) > * *contrib/prometheus-exporter*: passing with ignores (not solid*) > * *contrib/velocity*: passing with ignores (not solid*) > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155836#comment-17155836
]
Cao Manh Dat commented on SOLR-14354:
-
Thank you [~erickerickson], I gonna push a fix for this soon. I should be more
careful on cherry-pick changes.
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result result) {
> try {
>
[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155825#comment-17155825
]
Erick Erickson commented on SOLR-14354:
---
[~caomanhdat] This is failing 8x compilations on Jenkins with an error message:
{code}
Build Log:
[...truncated 12441 lines...]
[javac] Compiling 765 source files to
C:\Users\jenkins\workspace\Lucene-Solr-8.x-Windows\solr\build\solr-solrj\classes\java
[javac]
C:\Users\jenkins\workspace\Lucene-Solr-8.x-Windows\solr\solrj\src\java\org\apache\solr\client\solrj\impl\LBHttp2SolrClient.java:151:
error: cannot infer type arguments for AsyncListener
[javac] return
((Http2SolrClient)getClient(baseUrl)).asyncRequest(req.getRequest(), null, new
AsyncListener<>() {
[javac]
^
[javac] reason: '<>' with anonymous inner classes is not supported in
-source 8
[javac] (use -source 9 or higher to enable '<>' with anonymous inner
classes)
[javac] where T is a type-variable:
{code}
Not sure how to fix it, this is the awkwardness with trunk and 8x being on
different Java versions
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155810#comment-17155810 ] Mark Robert Miller commented on SOLR-14636: --- Finding old ghosts I had totally forgotten - I knew I hated that currency field, but I couldn’t remember of peep of why. Dynamic field concurrency bugs. Not a fun one, maybe punt while I’m playing in ez mode. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul resolved SOLR-14404.
---
Fix Version/s: 8.6
Resolution: Fixed
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
> Components: packages
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Fix For: 8.6
>
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "full.ClassName",
> "path-prefix" : "some-path-prefix"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "pkgName:full.ClassName" ,
> "path-prefix" : "some-path-prefix" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName", "path-prefix" : "some-path-prefix" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/$path-prefix/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at
> {{/api/some-path-prefix/path1}}. It's possible to add more methods at
> different paths. Ensure that all paths start with {{$path-prefix}} because
> that is the prefix in which the plugin is registered with. So
> {{/some-path-prefix/path2}} , {{/some-path-prefix/my/deeply/nested/path}} are
> all valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{/cluster/some/other/path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noble Paul updated SOLR-14404:
--
Component/s: packages
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
> Components: packages
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "full.ClassName",
> "path-prefix" : "some-path-prefix"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "pkgName:full.ClassName" ,
> "path-prefix" : "some-path-prefix" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName", "path-prefix" : "some-path-prefix" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/$path-prefix/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at
> {{/api/some-path-prefix/path1}}. It's possible to add more methods at
> different paths. Ensure that all paths start with {{$path-prefix}} because
> that is the prefix in which the plugin is registered with. So
> {{/some-path-prefix/path2}} , {{/some-path-prefix/my/deeply/nested/path}} are
> all valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{/cluster/some/other/path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13101) Shared storage support in SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155802#comment-17155802 ] Noble Paul commented on SOLR-13101: --- [~andy_vuong]Vuong is it anything that stops us from making this a plugin available through Solr's package system? Making it as a package helps many more users to use it and make the feedback loop faster > Shared storage support in SolrCloud > --- > > Key: SOLR-13101 > URL: https://issues.apache.org/jira/browse/SOLR-13101 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > > Solr should have first-class support for shared storage (blob/object stores > like S3, google cloud storage, etc. and shared filesystems like HDFS, NFS, > etc). > The key component will likely be a new replica type for shared storage. It > would have many of the benefits of the current "pull" replicas (not indexing > on all replicas, all shards identical with no shards getting out-of-sync, > etc), but would have additional benefits: > - Any shard could become leader (the blob store always has the index) > - Better elasticity scaling down >- durability not linked to number of replcias.. a single replica could be > common for write workloads >- could drop to 0 replicas for a shard when not needed (blob store always > has index) > - Allow for higher performance write workloads by skipping the transaction > log >- don't pay for what you don't need >- a commit will be necessary to flush to stable storage (blob store) > - A lot of the complexity and failure modes go away > An additional component a Directory implementation that will work well with > blob stores. We probably want one that treats local disk as a cache since > the latency to remote storage is so large. I think there are still some > "locking" issues to be solved here (ensuring that more than one writer to the > same index won't corrupt it). This should probably be pulled out into a > different JIRA issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155774#comment-17155774 ] Mark Robert Miller commented on SOLR-14636: --- I feel like Marcus Lemonis, "you just follow the process." Zoom, zoom, zoom, and we can simply grind the gremlins right out of this. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13205) StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
[
https://issues.apache.org/jira/browse/SOLR-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155717#comment-17155717
]
Pramod Kumar edited comment on SOLR-13205 at 7/10/20, 9:00 PM:
---
Hey Jason,
The change in init that checks the default field for emptiness is partly
related. That check will ensure that a useless default field is not passed in,
and we can error out early.
The parser checks I ended up doing since I was touching that method anyway.
Regarding CHANGES.txt, my alias (pramodkumar9) please!
Thanks for taking a peek Jason.
Pramod
was (Author: pramodkum...@yahoo.com):
Hey Jason,
The change in init that checks the default field for emptiness is partly
related. That check will ensure that a useless default field is not passed in,
and we can error out early.
The parser checks I ended up doing since I was touching that method anyway.
Regarding CHANGES.txt, my alias (pramodkumar9) please!
Thanks for taking a peek Jason.
Pramod
Not sure
> StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
>
>
> Key: SOLR-13205
> URL: https://issues.apache.org/jira/browse/SOLR-13205
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection and reproducing the bug
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html].
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> curl -v “URL_BUG”
> {noformat}
> Please check the issue description below to find the “URL_BUG” that will
> allow you to reproduce the issue reported.
>Reporter: Johannes Kloos
>Assignee: Jason Gerlowski
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13205.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?df==debug=all=on
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: 0
> at java.lang.String.charAt(String.java:658)
> at
> org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1045)
> at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:801)
> at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
> at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
> at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
> at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
> at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:255)
> at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
> at org.apache.solr.search.QParser.getQuery(QParser.java:173)
> at
> org.apache.solr.util.SolrPluginUtils.doSimpleQuery(SolrPluginUtils.java:479)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardResultsDebug(SolrPluginUtils.java:390)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardDebug(SolrPluginUtils.java:343)
> at
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:100)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:306)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
>
[jira] [Commented] (SOLR-13205) StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
[
https://issues.apache.org/jira/browse/SOLR-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155717#comment-17155717
]
Pramod Kumar commented on SOLR-13205:
-
Hey Jason,
The change in init that checks the default field for emptiness is partly
related. That check will ensure that a useless default field is not passed in,
and we can error out early.
The parser checks I ended up doing since I was touching that method anyway.
Regarding CHANGES.txt, my alias (pramodkumar9) please!
Thanks for taking a peek Jason.
Pramod
Not sure
> StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
>
>
> Key: SOLR-13205
> URL: https://issues.apache.org/jira/browse/SOLR-13205
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection and reproducing the bug
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html].
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> curl -v “URL_BUG”
> {noformat}
> Please check the issue description below to find the “URL_BUG” that will
> allow you to reproduce the issue reported.
>Reporter: Johannes Kloos
>Assignee: Jason Gerlowski
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13205.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?df==debug=all=on
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: 0
> at java.lang.String.charAt(String.java:658)
> at
> org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1045)
> at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:801)
> at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
> at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
> at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
> at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
> at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:255)
> at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
> at org.apache.solr.search.QParser.getQuery(QParser.java:173)
> at
> org.apache.solr.util.SolrPluginUtils.doSimpleQuery(SolrPluginUtils.java:479)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardResultsDebug(SolrPluginUtils.java:390)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardDebug(SolrPluginUtils.java:343)
> at
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:100)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:306)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> {noformat}
> The “df” parameter means that we set the empty string as default field(!).
> Since we do not give a field in the query, the default field is substituted
> in getFieldQuery. We then check if the field starts with “_” by using
> charAt(0).
> A trivial fix would be to replace field.charAt(0) == ‘_’ with
> field.startsWith(“_”).
> To set up an environment to reproduce this bug, follow the description in the
> ‘Environment’ field.
> We
[jira] [Commented] (SOLR-13205) StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
[
https://issues.apache.org/jira/browse/SOLR-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155706#comment-17155706
]
Jason Gerlowski commented on SOLR-13205:
Hey Pramod - got a chance to review your patch this afternoon. It all looks
great - with one partial exception: I don't fully understand why you made the
changes you did to {{SolrQueryParserBase.init}}. Are those tied in some way to
the IOOBE this patch resolves, or were you just improving things while you were
in the file? The two null/empty-check conditionals there seem like generally
good defensive coding, so I'm fine with them as long as the tests pass. Just
wanted to make sure I understood them in case there's something I'm missing.
Plan on running the tests and then committing this, as long as I hear back from
you with confirmation about that logic.
One last question - do you have a specific name/username you would like
credited as in the solr/CHANGES.txt file? I'll credit you as your JIRA name
"Pramod Kumar" unless you want something else in particular. Some people are
particular about it, so it always pays to ask.
> StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
>
>
> Key: SOLR-13205
> URL: https://issues.apache.org/jira/browse/SOLR-13205
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection and reproducing the bug
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html].
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> curl -v “URL_BUG”
> {noformat}
> Please check the issue description below to find the “URL_BUG” that will
> allow you to reproduce the issue reported.
>Reporter: Johannes Kloos
>Assignee: Jason Gerlowski
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13205.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?df==debug=all=on
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: 0
> at java.lang.String.charAt(String.java:658)
> at
> org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1045)
> at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:801)
> at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
> at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
> at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
> at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
> at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:255)
> at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
> at org.apache.solr.search.QParser.getQuery(QParser.java:173)
> at
> org.apache.solr.util.SolrPluginUtils.doSimpleQuery(SolrPluginUtils.java:479)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardResultsDebug(SolrPluginUtils.java:390)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardDebug(SolrPluginUtils.java:343)
> at
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:100)
> at
>
[jira] [Assigned] (SOLR-13205) StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
[
https://issues.apache.org/jira/browse/SOLR-13205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gerlowski reassigned SOLR-13205:
--
Assignee: Jason Gerlowski
> StringIndexOutOfBoundsException in SolrQueryParserBase.getFieldQuery
>
>
> Key: SOLR-13205
> URL: https://issues.apache.org/jira/browse/SOLR-13205
> Project: Solr
> Issue Type: Bug
> Components: search
>Affects Versions: master (9.0)
> Environment: h1. Steps to reproduce
> * Use a Linux machine.
> * Build commit {{ea2c8ba}} of Solr as described in the section below.
> * Build the films collection as described below.
> * Start the server using the command {{./bin/solr start -f -p 8983 -s
> /tmp/home}}
> * Request the URL given in the bug description.
> h1. Compiling the server
> {noformat}
> git clone https://github.com/apache/lucene-solr
> cd lucene-solr
> git checkout ea2c8ba
> ant compile
> cd solr
> ant server
> {noformat}
> h1. Building the collection and reproducing the bug
> We followed [Exercise
> 2|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html#exercise-2] from
> the [Solr
> Tutorial|http://lucene.apache.org/solr/guide/7_5/solr-tutorial.html].
> {noformat}
> mkdir -p /tmp/home
> echo '' >
> /tmp/home/solr.xml
> {noformat}
> In one terminal start a Solr instance in foreground:
> {noformat}
> ./bin/solr start -f -p 8983 -s /tmp/home
> {noformat}
> In another terminal, create a collection of movies, with no shards and no
> replication, and initialize it:
> {noformat}
> bin/solr create -c films
> curl -X POST -H 'Content-type:application/json' --data-binary '{"add-field":
> {"name":"name", "type":"text_general", "multiValued":false, "stored":true}}'
> http://localhost:8983/solr/films/schema
> curl -X POST -H 'Content-type:application/json' --data-binary
> '{"add-copy-field" : {"source":"*","dest":"_text_"}}'
> http://localhost:8983/solr/films/schema
> ./bin/post -c films example/films/films.json
> curl -v “URL_BUG”
> {noformat}
> Please check the issue description below to find the “URL_BUG” that will
> allow you to reproduce the issue reported.
>Reporter: Johannes Kloos
>Assignee: Jason Gerlowski
>Priority: Minor
> Labels: diffblue, newdev
> Attachments: SOLR-13205.patch
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Requesting the following URL causes Solr to return an HTTP 500 error response:
> {noformat}
> http://localhost:8983/solr/films/select?df==debug=all=on
> {noformat}
> The error response seems to be caused by the following uncaught exception:
> {noformat}
> java.lang.StringIndexOutOfBoundsException: String index out of range: 0
> at java.lang.String.charAt(String.java:658)
> at
> org.apache.solr.parser.SolrQueryParserBase.getFieldQuery(SolrQueryParserBase.java:1045)
> at
> org.apache.solr.parser.SolrQueryParserBase.handleBareTokenQuery(SolrQueryParserBase.java:801)
> at org.apache.solr.parser.QueryParser.Term(QueryParser.java:421)
> at org.apache.solr.parser.QueryParser.Clause(QueryParser.java:278)
> at org.apache.solr.parser.QueryParser.Query(QueryParser.java:162)
> at org.apache.solr.parser.QueryParser.TopLevelQuery(QueryParser.java:131)
> at
> org.apache.solr.parser.SolrQueryParserBase.parse(SolrQueryParserBase.java:255)
> at org.apache.solr.search.LuceneQParser.parse(LuceneQParser.java:49)
> at org.apache.solr.search.QParser.getQuery(QParser.java:173)
> at
> org.apache.solr.util.SolrPluginUtils.doSimpleQuery(SolrPluginUtils.java:479)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardResultsDebug(SolrPluginUtils.java:390)
> at
> org.apache.solr.util.SolrPluginUtils.doStandardDebug(SolrPluginUtils.java:343)
> at
> org.apache.solr.handler.component.DebugComponent.process(DebugComponent.java:100)
> at
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:306)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
> {noformat}
> The “df” parameter means that we set the empty string as default field(!).
> Since we do not give a field in the query, the default field is substituted
> in getFieldQuery. We then check if the field starts with “_” by using
> charAt(0).
> A trivial fix would be to replace field.charAt(0) == ‘_’ with
> field.startsWith(“_”).
> To set up an environment to reproduce this bug, follow the description in the
> ‘Environment’ field.
> We automatically found this issue and ~70 more like this using [Diffblue
> Microservices Testing|https://www.diffblue.com/labs/?utm_source=solr-br].
> Find more information on this [fuzz testing
> campaign|https://www.diffblue.com/blog/2018/12/19/diffblue-microservice-testing-a-sneak-peek-at-our-early-product-and-results?utm_source=solr-br].
--
This message was sent by Atlassian
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155658#comment-17155658 ] Mark Robert Miller commented on SOLR-14636: --- I couldn't get this thing to work for crap for 10 years and now I can send it to the moon and back if I want - some excitement along the trail might leak out, but it won't affect the results ;) Those will be very scientific. Very scientific. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13101) Shared storage support in SolrCloud
[ https://issues.apache.org/jira/browse/SOLR-13101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155639#comment-17155639 ] Andy Vuong commented on SOLR-13101: --- Hey Solr Community! I wanted to share an update on this JIRA. We've recently decided to continue work on this project internally for convenience and as we work through new challenges and re-visit our design of shared storage. Some of these challenges include: * Forcing commits and pushes to Blob on each (sub) indexing batch makes things expensive (paying traffic to S3) and less efficient from a SolrCloud perspective (too many small commits, merge cost), * Delaying ack to client on an indexing batch until data is indexed, segment is created then pushed to S3 slows things down considerably, * Transaction logs are used heavily in SolrCloud code. Having nodes with non persistent storage is challenging (for example post shard split recovery mode) Work is progressing but we'll no longer use our feature branch for this work. We'll be sure to keep the community updated in the future as we progress on addressing these issues. > Shared storage support in SolrCloud > --- > > Key: SOLR-13101 > URL: https://issues.apache.org/jira/browse/SOLR-13101 > Project: Solr > Issue Type: New Feature > Components: SolrCloud >Reporter: Yonik Seeley >Priority: Major > Time Spent: 15h 50m > Remaining Estimate: 0h > > Solr should have first-class support for shared storage (blob/object stores > like S3, google cloud storage, etc. and shared filesystems like HDFS, NFS, > etc). > The key component will likely be a new replica type for shared storage. It > would have many of the benefits of the current "pull" replicas (not indexing > on all replicas, all shards identical with no shards getting out-of-sync, > etc), but would have additional benefits: > - Any shard could become leader (the blob store always has the index) > - Better elasticity scaling down >- durability not linked to number of replcias.. a single replica could be > common for write workloads >- could drop to 0 replicas for a shard when not needed (blob store always > has index) > - Allow for higher performance write workloads by skipping the transaction > log >- don't pay for what you don't need >- a commit will be necessary to flush to stable storage (blob store) > - A lot of the complexity and failure modes go away > An additional component a Directory implementation that will work well with > blob stores. We probably want one that treats local disk as a cache since > the latency to remote storage is so large. I think there are still some > "locking" issues to be solved here (ensuring that more than one writer to the > same index won't corrupt it). This should probably be pulled out into a > different JIRA issue. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-11208) Usage SynchronousQueue in Executors prevent large scale operations
[
https://issues.apache.org/jira/browse/SOLR-11208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155640#comment-17155640
]
Ilan Ginzburg commented on SOLR-11208:
--
I've put out a PR [https://github.com/apache/lucene-solr/pull/1664].
Tests pass. Would like somebody to take a look in case I missed something.
> Usage SynchronousQueue in Executors prevent large scale operations
> --
>
> Key: SOLR-11208
> URL: https://issues.apache.org/jira/browse/SOLR-11208
> Project: Solr
> Issue Type: Bug
>Affects Versions: 6.6
>Reporter: Björn Häuser
>Priority: Major
> Attachments: response.json
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I am not sure where to start with this one.
> I tried to post this already on the mailing list:
> https://mail-archives.apache.org/mod_mbox/lucene-solr-user/201708.mbox/%3c48c49426-33a2-4d79-ae26-a4515b8f8...@gmail.com%3e
> In short: the usage of a SynchronousQueue as the workQeue prevents more tasks
> than max threads.
> For example, taken from OverseerCollectionMessageHandler:
> {code:java}
> ExecutorService tpe = new ExecutorUtil.MDCAwareThreadPoolExecutor(5, 10,
> 0L, TimeUnit.MILLISECONDS,
> new SynchronousQueue<>(),
> new
> DefaultSolrThreadFactory("OverseerCollectionMessageHandlerThreadFactory"));
> {code}
> This Executor is used when doing a REPLACENODE (= ADDREPLICA) command. When
> the node has more than 10 collections this will fail with the mentioned
> java.util.concurrent.RejectedExecutionException.
> I am also not sure how to fix this. Just replacing the queue with a different
> implementation feels wrong to me or could cause unwanted side behaviour.
> Thanks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14643) Edit Ref Guide about How to Configure Tokenizing Managed Synonyms
[
https://issues.apache.org/jira/browse/SOLR-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kayak28 updated SOLR-14643:
---
Summary: Edit Ref Guide about How to Configure Tokenizing Managed Synonyms
(was: Edit Ref Guide about how to configure Tokenizing Managed Synonyms)
> Edit Ref Guide about How to Configure Tokenizing Managed Synonyms
> -
>
> Key: SOLR-14643
> URL: https://issues.apache.org/jira/browse/SOLR-14643
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: kayak28
>Priority: Minor
>
> SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
> synonyms in text files.
> The configuration is below:
>
> {{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
> {{}}{{ignoreCase}}{{=}}{{"true"}}
> {{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
> {{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
> For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
> Managed Resource JSON file, there does not exists any explanation about
> tokenizer configuration.
> For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference
> guide should contain the configuration guide.
>
> When a user want to tokenize their synonyms, which is used by
> ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON
> like below.
> { "responseHeader":\\{ "status":0, "QTime":3}
> ,
> "synonymMappings":{ "
> initArgs":
> { "ignoreCase":true, "format":"solr", {color:#ffab00}
> "tokenizerFactory":"solr.Factory" {color} }
> ,
> "initializedOn":"2014-12-16T22:44:05.33Z",
> "managedMap":
> { "GB": ["GiB", "Gigabyte"], "TV": ["Television"],
> "happy": ["glad", "joyful"]}
> }}
>
> Also, it might be nice if we could edit Managed Resource JSON via REST API.
> (This could be another issue.)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14643) Edit Ref Guide about how to configure Tokenizing Managed Synonyms
[
https://issues.apache.org/jira/browse/SOLR-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kayak28 updated SOLR-14643:
---
Priority: Minor (was: Major)
> Edit Ref Guide about how to configure Tokenizing Managed Synonyms
> -
>
> Key: SOLR-14643
> URL: https://issues.apache.org/jira/browse/SOLR-14643
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: kayak28
>Priority: Minor
>
> SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
> synonyms in text files.
> The configuration is below:
>
> {{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
> {{}}{{ignoreCase}}{{=}}{{"true"}}
> {{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
> {{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
> For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
> Managed Resource JSON file, there does not exists any explanation about
> tokenizer configuration.
> For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference
> guide should contain the configuration guide.
>
> When a user want to tokenize their synonyms, which is used by
> ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON
> like below.
> { "responseHeader":\\{ "status":0, "QTime":3}
> ,
> "synonymMappings":{ "
> initArgs":
> { "ignoreCase":true, "format":"solr", {color:#ffab00}
> "tokenizerFactory":"solr.Factory" {color} }
> ,
> "initializedOn":"2014-12-16T22:44:05.33Z",
> "managedMap":
> { "GB": ["GiB", "Gigabyte"], "TV": ["Television"],
> "happy": ["glad", "joyful"]}
> }}
>
> Also, it might be nice if we could edit Managed Resource JSON via REST API.
> (This could be another issue.)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14643) Edit Ref Guide about how to configure Tokenizing Managed Synonyms
[
https://issues.apache.org/jira/browse/SOLR-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kayak28 updated SOLR-14643:
---
Description:
SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
synonyms in text files.
The configuration is below:
{{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
{{}}{{ignoreCase}}{{=}}{{"true"}}
{{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
{{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
Managed Resource JSON file, there does not exists any explanation about
tokenizer configuration.
For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference guide
should contain the configuration guide.
When a user want to tokenize their synonyms, which is used by
ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON like
below.
{ "responseHeader":\{ "status":0, "QTime":3},
"synonymMappings":{ "
initArgs":{
"ignoreCase":true,
"format":"solr",
"tokenizerFactory":"solr.Factory" },
"initializedOn":"2014-12-16T22:44:05.33Z",
"managedMap":{
"GB": ["GiB", "Gigabyte"],
"TV": ["Television"],
"happy": ["glad", "joyful"]}
}}
Also, it might be nice if we could edit Managed Resource JSON via REST API.
(This could be another issue.)
was:
SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
synonyms in text files.
The configuration is below:
{{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
{{}}{{ignoreCase}}{{=}}{{"true"}}
{{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
{{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
{{}}
For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
Managed Resource JSON file, there does not exists any explanation about
tokenizer configuration.
For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference guide
should contain the configuration guide.
When a user want to tokenize their synonyms, which is used by
ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON like
below.
{ "responseHeader":{
"status":0, "QTime":3},
"synonymMappings":{ "
initArgs":{
"ignoreCase":true,
"format":"solr",
"tokenizerFactory":"solr.Factory"
},
"initializedOn":"2014-12-16T22:44:05.33Z",
"managedMap":{
"GB": ["GiB", "Gigabyte"],
"TV": ["Television"],
"happy": ["glad", "joyful"]}}}
Also, it might be nice if we could edit Managed Resource JSON via REST API.
(This could be another issue.)
{{}}
> Edit Ref Guide about how to configure Tokenizing Managed Synonyms
> -
>
> Key: SOLR-14643
> URL: https://issues.apache.org/jira/browse/SOLR-14643
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: kayak28
>Priority: Major
>
> SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
> synonyms in text files.
> The configuration is below:
>
> {{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
> {{}}{{ignoreCase}}{{=}}{{"true"}}
> {{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
> {{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
> For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
> Managed Resource JSON file, there does not exists any explanation about
> tokenizer configuration.
> For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference
> guide should contain the configuration guide.
>
> When a user want to tokenize their synonyms, which is used by
> ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON
> like below.
> { "responseHeader":\{ "status":0, "QTime":3},
> "synonymMappings":{ "
> initArgs":{
> "ignoreCase":true,
> "format":"solr",
> "tokenizerFactory":"solr.Factory" },
> "initializedOn":"2014-12-16T22:44:05.33Z",
> "managedMap":{
> "GB": ["GiB", "Gigabyte"],
> "TV": ["Television"],
> "happy": ["glad", "joyful"]}
> }}
>
> Also, it might be nice if we could edit Managed Resource JSON via REST API.
> (This could be another issue.)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14643) Edit Ref Guide about how to configure Tokenizing Managed Synonyms
[
https://issues.apache.org/jira/browse/SOLR-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kayak28 updated SOLR-14643:
---
Description:
SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
synonyms in text files.
The configuration is below:
{{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
{{}}{{ignoreCase}}{{=}}{{"true"}}
{{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
{{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
Managed Resource JSON file, there does not exists any explanation about
tokenizer configuration.
For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference guide
should contain the configuration guide.
When a user want to tokenize their synonyms, which is used by
ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON like
below.
{ "responseHeader":\\{ "status":0, "QTime":3}
,
"synonymMappings":{ "
initArgs":
{ "ignoreCase":true, "format":"solr", {color:#ffab00}
"tokenizerFactory":"solr.Factory" {color} }
,
"initializedOn":"2014-12-16T22:44:05.33Z",
"managedMap":
{ "GB": ["GiB", "Gigabyte"], "TV": ["Television"],
"happy": ["glad", "joyful"]}
}}
Also, it might be nice if we could edit Managed Resource JSON via REST API.
(This could be another issue.)
was:
SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
synonyms in text files.
The configuration is below:
{{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
{{}}{{ignoreCase}}{{=}}{{"true"}}
{{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
{{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
Managed Resource JSON file, there does not exists any explanation about
tokenizer configuration.
For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference guide
should contain the configuration guide.
When a user want to tokenize their synonyms, which is used by
ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON like
below.
{ "responseHeader":\{ "status":0, "QTime":3},
"synonymMappings":{ "
initArgs":{
"ignoreCase":true,
"format":"solr",
"tokenizerFactory":"solr.Factory" },
"initializedOn":"2014-12-16T22:44:05.33Z",
"managedMap":{
"GB": ["GiB", "Gigabyte"],
"TV": ["Television"],
"happy": ["glad", "joyful"]}
}}
Also, it might be nice if we could edit Managed Resource JSON via REST API.
(This could be another issue.)
> Edit Ref Guide about how to configure Tokenizing Managed Synonyms
> -
>
> Key: SOLR-14643
> URL: https://issues.apache.org/jira/browse/SOLR-14643
> Project: Solr
> Issue Type: Task
> Security Level: Public(Default Security Level. Issues are Public)
> Components: documentation
>Reporter: kayak28
>Priority: Major
>
> SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
> synonyms in text files.
> The configuration is below:
>
> {{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
> {{}}{{ignoreCase}}{{=}}{{"true"}}
> {{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
> {{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
> For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
> Managed Resource JSON file, there does not exists any explanation about
> tokenizer configuration.
> For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference
> guide should contain the configuration guide.
>
> When a user want to tokenize their synonyms, which is used by
> ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON
> like below.
> { "responseHeader":\\{ "status":0, "QTime":3}
> ,
> "synonymMappings":{ "
> initArgs":
> { "ignoreCase":true, "format":"solr", {color:#ffab00}
> "tokenizerFactory":"solr.Factory" {color} }
> ,
> "initializedOn":"2014-12-16T22:44:05.33Z",
> "managedMap":
> { "GB": ["GiB", "Gigabyte"], "TV": ["Television"],
> "happy": ["glad", "joyful"]}
> }}
>
> Also, it might be nice if we could edit Managed Resource JSON via REST API.
> (This could be another issue.)
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail:
[jira] [Created] (SOLR-14643) Edit Ref Guide about how to configure Tokenizing Managed Synonyms
kayak28 created SOLR-14643:
--
Summary: Edit Ref Guide about how to configure Tokenizing Managed
Synonyms
Key: SOLR-14643
URL: https://issues.apache.org/jira/browse/SOLR-14643
Project: Solr
Issue Type: Task
Security Level: Public (Default Security Level. Issues are Public)
Components: documentation
Reporter: kayak28
SynonymGraphFilter and SynonymFileter can configure a tokenizer to tokenize
synonyms in text files.
The configuration is below:
{{<}}{{filter}} {{class}}{{=}}{{"solr.SynonymGraphFilterFactory"}}
{{}}{{ignoreCase}}{{=}}{{"true"}}
{{}}{{synonyms}}{{=}}{{"synonyms.txt"}}
{{}}{{tokenizerFactory}}{{=}}{{"solr.[Name of Tokenizer]"}}{{/>}}
{{}}
For ManagedSynonymGraphFilter, even though it can configure tokenizer in the
Managed Resource JSON file, there does not exists any explanation about
tokenizer configuration.
For users to choose ManagedSynonymGraphFilter seamlessly, Solr reference guide
should contain the configuration guide.
When a user want to tokenize their synonyms, which is used by
ManagedSynonymGraphFilter, then the user should edit Managed Resource JSON like
below.
{ "responseHeader":{
"status":0, "QTime":3},
"synonymMappings":{ "
initArgs":{
"ignoreCase":true,
"format":"solr",
"tokenizerFactory":"solr.Factory"
},
"initializedOn":"2014-12-16T22:44:05.33Z",
"managedMap":{
"GB": ["GiB", "Gigabyte"],
"TV": ["Television"],
"happy": ["glad", "joyful"]}}}
Also, it might be nice if we could edit Managed Resource JSON via REST API.
(This could be another issue.)
{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155610#comment-17155610 ] Mark Robert Miller commented on SOLR-14636: --- Yeah baby, if this was an orchestra, the solr core test module has started to sing on key over half the time. That module's gonna be a superstar. > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] murblanc opened a new pull request #1664: SOLR-11208: Usage SynchronousQueue in Executors prevent large scale operations
murblanc opened a new pull request #1664: URL: https://github.com/apache/lucene-solr/pull/1664 In `OverseerCollectionMessageHandler` replace the `SynchronousQueue` by a `LinkedBlockingQueue` for handing more jobs to the executor service than there are threads. Given this is an old Jira, the issue is real, the OP suggested that easy fix almost 3 years ago and that nobody implemented it, there might be something I missed here, so will not push that code before somebody reviews it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14597) Advanced Query Parser
[
https://issues.apache.org/jira/browse/SOLR-14597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155587#comment-17155587
]
Gus Heck commented on SOLR-14597:
-
After some work came up with this which omits files that don't have "java" in
their name, but should give a decent idea:
{code:java}
NS2-MacBook-Pro:lucene-solr-cdg3 gus$ git diff HEAD..master_head | grep 'diff
..git' | grep java |sed 's#b/#@#' | rev | cut -d'@' -f 1 | rev
gradle/generation/javacc.gradle
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/DropIfFlaggedFilter.java
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/DropIfFlaggedFilterFactory.java
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/PatternTypingFilter.java
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/PatternTypingFilterFactory.java
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TypeAsSynonymFilter.java
lucene/analysis/common/src/java/org/apache/lucene/analysis/miscellaneous/TypeAsSynonymFilterFactory.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/minhash/MinHashFilterTest.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestConcatenatingTokenStream.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestDropIfFlaggedFilter.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestDropIfFlaggedFilterFactory.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestPatternTypingFilter.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestPatternTypingFilterFactory.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestTypeAsSynonymFilter.java
lucene/analysis/common/src/test/org/apache/lucene/analysis/miscellaneous/TestTypeAsSynonymFilterFactory.java
lucene/core/src/test/org/apache/lucene/analysis/TestStopFilter.java
lucene/test-framework/src/java/org/apache/lucene/analysis/BaseTokenStreamTestCase.java
solr/core/src/java/org/apache/solr/analysis/TokenAnalyzerFilter.java
solr/core/src/java/org/apache/solr/analysis/TokenAnalyzerFilterFactory.java
solr/core/src/java/org/apache/solr/aqp/AdvToken.java
solr/core/src/java/org/apache/solr/aqp/AdvancedQueryParserBase.java
solr/core/src/java/org/apache/solr/aqp/ParseException.java
solr/core/src/java/org/apache/solr/aqp/QueryParser.java
solr/core/src/java/org/apache/solr/aqp/QueryParser.jj
solr/core/src/java/org/apache/solr/aqp/QueryParserConstants.java
solr/core/src/java/org/apache/solr/aqp/QueryParserTokenManager.java
solr/core/src/java/org/apache/solr/aqp/SpanContext.java
solr/core/src/java/org/apache/solr/aqp/Token.java
solr/core/src/java/org/apache/solr/aqp/TokenMgrError.java
solr/core/src/java/org/apache/solr/aqp/package-info.java
solr/core/src/java/org/apache/solr/parser/Operator.java
solr/core/src/java/org/apache/solr/parser/QueryParser.java
solr/core/src/java/org/apache/solr/parser/QueryParser.jj
solr/core/src/java/org/apache/solr/parser/SolrQueryParserBase.java
solr/core/src/java/org/apache/solr/parser/SynonymQueryStyle.java
solr/core/src/java/org/apache/solr/schema/IndexSchema.java
solr/core/src/java/org/apache/solr/schema/TextField.java
solr/core/src/java/org/apache/solr/search/AdvancedQParser.java
solr/core/src/java/org/apache/solr/search/AdvancedQParserPlugin.java
solr/core/src/java/org/apache/solr/search/AdvancedQueryParser.java
solr/core/src/java/org/apache/solr/search/ComplexPhraseQParserPlugin.java
solr/core/src/java/org/apache/solr/search/DisMaxQParser.java
solr/core/src/java/org/apache/solr/search/ExtendedDismaxQParser.java
solr/core/src/java/org/apache/solr/search/QParserPlugin.java
solr/core/src/java/org/apache/solr/search/QueryParsing.java
solr/core/src/java/org/apache/solr/search/SimpleQParserPlugin.java
solr/core/src/java/org/apache/solr/util/SolrPluginUtils.java
solr/core/src/test/org/apache/solr/analysis/PatternTypingFilterFactoryTest.java
solr/core/src/test/org/apache/solr/analysis/TokenAnalyzerFilterFactoryTest.java
solr/core/src/test/org/apache/solr/aqp/AbstractAqpTestCase.java
solr/core/src/test/org/apache/solr/aqp/CharacterRangeTest.java
solr/core/src/test/org/apache/solr/aqp/FieldedSearchTest.java
solr/core/src/test/org/apache/solr/aqp/LiteralPhraseTest.java
solr/core/src/test/org/apache/solr/aqp/MustNotTest.java
solr/core/src/test/org/apache/solr/aqp/MustTest.java
solr/core/src/test/org/apache/solr/aqp/NumericSearchTest.java
solr/core/src/test/org/apache/solr/aqp/OrderedDistanceGroupTest.java
solr/core/src/test/org/apache/solr/aqp/PhraseTest.java
solr/core/src/test/org/apache/solr/aqp/ShouldTest.java
solr/core/src/test/org/apache/solr/aqp/SimpleGroupTest.java
solr/core/src/test/org/apache/solr/aqp/SimpleQueryTest.java
solr/core/src/test/org/apache/solr/aqp/TemporalFieldedSearchTest.java
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Attachment: (was: solr-core-serial-run.gif) > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Description: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] *status*: alpha *speed*: ludicrous *tests***: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ _** Non Nightly currently, Nightly comes last._ was: SolrCloud powers critical infrastructure and needs the ability to run quickly with stability. This reference implementation will allow for this. *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] *status*: alpha *tests***: * *core*: passing with ignores (not solid*) * *solrj*: tbd * *test-framework*: tbd * *contrib/analysis-extras*: tbd * *contrib/analytics*: tbd * *contrib/clustering*: tbd * *contrib/dataimporthandler*: tbd * *contrib/dataimporthandler-extras*: tbd * *contrib/extraction*: tbd * *contrib/jaegertracer-configurator*: tbd * *contrib/langid*: tbd * *contrib/prometheus-exporter*: tbd * *contrib/velocity*: tbd _* Running tests quickly and efficiently with strict policing will more frequently find bugs and requires a period of hardening._ _** Non Nightly currently, Nightly comes last._ > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Issue Comment Deleted] (SOLR-14636) Provide a reference implementation for SolrCloud that is stable and fast.
[ https://issues.apache.org/jira/browse/SOLR-14636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mark Robert Miller updated SOLR-14636: -- Comment: was deleted (was: !solr-core-serial-run.gif!) > Provide a reference implementation for SolrCloud that is stable and fast. > - > > Key: SOLR-14636 > URL: https://issues.apache.org/jira/browse/SOLR-14636 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Mark Robert Miller >Assignee: Mark Robert Miller >Priority: Major > > SolrCloud powers critical infrastructure and needs the ability to run quickly > with stability. This reference implementation will allow for this. > *location*: [https://github.com/apache/lucene-solr/tree/reference_impl] > *status*: alpha > *speed*: ludicrous > *tests***: > * *core*: passing with ignores (not solid*) > * *solrj*: tbd > * *test-framework*: tbd > * *contrib/analysis-extras*: tbd > * *contrib/analytics*: tbd > * *contrib/clustering*: tbd > * *contrib/dataimporthandler*: tbd > * *contrib/dataimporthandler-extras*: tbd > * *contrib/extraction*: tbd > * *contrib/jaegertracer-configurator*: tbd > * *contrib/langid*: tbd > * *contrib/prometheus-exporter*: tbd > * *contrib/velocity*: tbd > _* Running tests quickly and efficiently with strict policing will more > frequently find bugs and requires a period of hardening._ > _** Non Nightly currently, Nightly comes last._ -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14642) Add a no-sort option for the /export handler
Joel Bernstein created SOLR-14642: - Summary: Add a no-sort option for the /export handler Key: SOLR-14642 URL: https://issues.apache.org/jira/browse/SOLR-14642 Project: Solr Issue Type: New Feature Security Level: Public (Default Security Level. Issues are Public) Reporter: Joel Bernstein The sort is the most expensive operation of the the /export handler. Spark-Solr relies heavily on the /export handler but doesn't make use of the sort. As Spark-Solr use continues to increase its time to add optimizations specifically for Spark-Solr. This ticket will add a no-sort option to the export handler to accelerate Spark-Solr queries. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14608) Faster sorting for the /export handler
[ https://issues.apache.org/jira/browse/SOLR-14608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155489#comment-17155489 ] ASF subversion and git services commented on SOLR-14608: Commit 95e706abc425003d79a037500b9887f2c8a7798c in lucene-solr's branch refs/heads/jira/SOLR-14608-export from Joel Bernstein [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=95e706a ] SOLR-14608: Size segment level sort queues based on segement maxdoc > Faster sorting for the /export handler > -- > > Key: SOLR-14608 > URL: https://issues.apache.org/jira/browse/SOLR-14608 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Joel Bernstein >Assignee: Andrzej Bialecki >Priority: Major > > The largest cost of the export handler is the sorting. This ticket will > implement an improved algorithm for sorting that should greatly increase > overall throughput for the export handler. > *The current algorithm is as follows:* > Collect a bitset of matching docs. Iterate over that bitset and materialize > the top level oridinals for the sort fields in the document and add them to > priority queue of size 3. Then export the top 3 docs, turn off the > bits in the bit set and iterate again until all docs are sorted and sent. > There are two performance bottlenecks with this approach: > 1) Materializing the top level ordinals adds a huge amount of overhead to the > sorting process. > 2) The size of priority queue, 30,000, adds significant overhead to sorting > operations. > *The new algorithm:* > Has a top level *merge sort iterator* that wraps segment level iterators that > perform segment level priority queue sorts. > *Segment level:* > The segment level docset will be iterated and the segment level ordinals for > the sort fields will be materialized and added to a segment level priority > queue. As the segment level iterator pops docs from the priority queue the > top level ordinals for the sort fields are materialized. Because the top > level ordinals are materialized AFTER the sort, they only need to be looked > up when the segment level ordinal changes. This takes advantage of the sort > to limit the lookups into the top level ordinal structures. This also > eliminates redundant lookups of top level ordinals that occur during the > multiple passes over the matching docset. > The segment level priority queues can be kept smaller than 30,000 to improve > performance of the sorting operations because the overall batch size will > still be 30,000 or greater when all the segment priority queue sizes are > added up. This allows for batch sizes much larger then 30,000 without using a > single large priority queue. The increased batch size means fewer iterations > over the matching docset and the decreased priority queue size means faster > sorting operations. > *Top level:* > A top level iterator does a merge sort over the segment level iterators by > comparing the top level ordinals materialized when the segment level docs are > popped from the segment level priority queues. This requires no extra memory > and will be very performant. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155073#comment-17155073
]
Noble Paul edited comment on SOLR-14404 at 7/10/20, 1:05 PM:
-
I have hardened the server code in the PR
[https://github.com/apache/lucene-solr/pull/1661]
I beasted it for some time. No failures yet. [~erickerickson] please try to
beast the branch it if possible
was (Author: noble.paul):
I have hardened the tests in the PR
[https://github.com/apache/lucene-solr/pull/1661]
I beasted it for some time. No failures yet. [~erickerickson] please try to
beast the branch it if possible
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "full.ClassName",
> "path-prefix" : "some-path-prefix"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "pkgName:full.ClassName" ,
> "path-prefix" : "some-path-prefix" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName", "path-prefix" : "some-path-prefix" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/$path-prefix/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at
> {{/api/some-path-prefix/path1}}. It's possible to add more methods at
> different paths. Ensure that all paths start with {{$path-prefix}} because
> that is the prefix in which the plugin is registered with. So
> {{/some-path-prefix/path2}} , {{/some-path-prefix/my/deeply/nested/path}} are
> all valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{/cluster/some/other/path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Resolved] (LUCENE-9321) Port documentation task to gradle
[ https://issues.apache.org/jira/browse/LUCENE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomoko Uchida resolved LUCENE-9321. --- Resolution: Resolved If I missed something, please feel free to reopen. > Port documentation task to gradle > - > > Key: LUCENE-9321 > URL: https://issues.apache.org/jira/browse/LUCENE-9321 > Project: Lucene - Core > Issue Type: Sub-task > Components: general/build >Reporter: Tomoko Uchida >Assignee: Uwe Schindler >Priority: Major > Fix For: master (9.0) > > Attachments: screenshot-1.png > > Time Spent: 9h > Remaining Estimate: 0h > > This is a placeholder issue for porting ant "documentation" task to gradle. > The generated documents should be able to be published on lucene.apache.org > web site on "as-is" basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9321) Port documentation task to gradle
[
https://issues.apache.org/jira/browse/LUCENE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155459#comment-17155459
]
Tomoko Uchida commented on LUCENE-9321:
---
We now have {{documentation}} task and its lint task {{checkBrokenLinks}} on
the master branch. I think we can resolve this.
> Port documentation task to gradle
> -
>
> Key: LUCENE-9321
> URL: https://issues.apache.org/jira/browse/LUCENE-9321
> Project: Lucene - Core
> Issue Type: Sub-task
> Components: general/build
>Reporter: Tomoko Uchida
>Assignee: Uwe Schindler
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: screenshot-1.png
>
> Time Spent: 9h
> Remaining Estimate: 0h
>
> This is a placeholder issue for porting ant "documentation" task to gradle.
> The generated documents should be able to be published on lucene.apache.org
> web site on "as-is" basis.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13132) Improve JSON "terms" facet performance when sorted by relatedness
[
https://issues.apache.org/jira/browse/SOLR-13132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155444#comment-17155444
]
Michael Gibney commented on SOLR-13132:
---
Excellent! Thanks [~hossman]!
> Improve JSON "terms" facet performance when sorted by relatedness
> --
>
> Key: SOLR-13132
> URL: https://issues.apache.org/jira/browse/SOLR-13132
> Project: Solr
> Issue Type: Improvement
> Components: Facet Module
>Affects Versions: 7.4, master (9.0)
>Reporter: Michael Gibney
>Assignee: Chris M. Hostetter
>Priority: Major
> Fix For: master (9.0), 8.7
>
> Attachments: SOLR-13132-benchmarks.tgz,
> SOLR-13132-with-cache-01.patch, SOLR-13132-with-cache.patch,
> SOLR-13132.patch, SOLR-13132_testSweep.patch
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> When sorting buckets by {{relatedness}}, JSON "terms" facet must calculate
> {{relatedness}} for every term.
> The current implementation uses a standard uninverted approach (either
> {{docValues}} or {{UnInvertedField}}) to get facet counts over the domain
> base docSet, and then uses that initial pass as a pre-filter for a
> second-pass, inverted approach of fetching docSets for each relevant term
> (i.e., {{count > minCount}}?) and calculating intersection size of those sets
> with the domain base docSet.
> Over high-cardinality fields, the overhead of per-term docSet creation and
> set intersection operations increases request latency to the point where
> relatedness sort may not be usable in practice (for my use case, even after
> applying the patch for SOLR-13108, for a field with ~220k unique terms per
> core, QTime for high-cardinality domain docSets were, e.g.: cardinality
> 1816684=9000ms, cardinality 5032902=18000ms).
> The attached patch brings the above example QTimes down to a manageable
> ~300ms and ~250ms respectively. The approach calculates uninverted facet
> counts over domain base, foreground, and background docSets in parallel in a
> single pass. This allows us to take advantage of the efficiencies built into
> the standard uninverted {{FacetFieldProcessorByArray[DV|UIF]}}), and avoids
> the per-term docSet creation and set intersection overhead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155403#comment-17155403
]
Erick Erickson commented on SOLR-14404:
---
OK, I'll start it out. Yesterday I only got 1 failure in 1,000 iterations so it
might be a while.
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "full.ClassName",
> "path-prefix" : "some-path-prefix"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "pkgName:full.ClassName" ,
> "path-prefix" : "some-path-prefix" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName", "path-prefix" : "some-path-prefix" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/$path-prefix/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at
> {{/api/some-path-prefix/path1}}. It's possible to add more methods at
> different paths. Ensure that all paths start with {{$path-prefix}} because
> that is the prefix in which the plugin is registered with. So
> {{/some-path-prefix/path2}} , {{/some-path-prefix/my/deeply/nested/path}} are
> all valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{/cluster/some/other/path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14469) Removed deprecated code in solr/core (master only)
[ https://issues.apache.org/jira/browse/SOLR-14469?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155400#comment-17155400 ] Erick Erickson commented on SOLR-14469: --- Oooh, thanks [~dsmiley]. I'd have just jumped in and missed that entirely. > Removed deprecated code in solr/core (master only) > -- > > Key: SOLR-14469 > URL: https://issues.apache.org/jira/browse/SOLR-14469 > Project: Solr > Issue Type: Sub-task >Reporter: Erick Erickson >Assignee: Erick Erickson >Priority: Major > > I'm currently working on getting all the warnings out of the code, so this is > something of a placeholder for a week or two. > There will be sub-tasks, please create them when you start working on a > project. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] CaoManhDat opened a new pull request #1663: SOLR-14641: PeerSync, remove canHandleVersionRanges check
CaoManhDat opened a new pull request #1663: URL: https://github.com/apache/lucene-solr/pull/1663 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9321) Port documentation task to gradle
[ https://issues.apache.org/jira/browse/LUCENE-9321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155253#comment-17155253 ] Dawid Weiss commented on LUCENE-9321: - Is there anything left we need to do here? > Port documentation task to gradle > - > > Key: LUCENE-9321 > URL: https://issues.apache.org/jira/browse/LUCENE-9321 > Project: Lucene - Core > Issue Type: Sub-task > Components: general/build >Reporter: Tomoko Uchida >Assignee: Uwe Schindler >Priority: Major > Fix For: master (9.0) > > Attachments: screenshot-1.png > > Time Spent: 9h > Remaining Estimate: 0h > > This is a placeholder issue for porting ant "documentation" task to gradle. > The generated documents should be able to be published on lucene.apache.org > web site on "as-is" basis. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (LUCENE-9077) Gradle build
[
https://issues.apache.org/jira/browse/LUCENE-9077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dawid Weiss updated LUCENE-9077:
Description:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on *gradle-master* branch. It is kept with sync with *master*.
Try running the following to see an overview of helper guides concerning
typical workflow, testing and ant-migration helpers:
gradlew :help
A list of items that needs to be added or requires work. If you'd like to work
on any of these, please add your name to the list. Once you have a patch/ pull
request let me (dweiss) know - I'll try to coordinate the merges.
* (/) Apply forbiddenAPIs
* (/) Generate hardware-aware gradle defaults for parallelism (count of
workers and test JVMs).
* (/) Fail the build if --tests filter is applied and no tests execute during
the entire build (this allows for an empty set of filtered tests at single
project level).
* (/) Port other settings and randomizations from common-build.xml
* (/) Configure security policy/ sandboxing for tests.
* (/) test's console output on -Ptests.verbose=true
* (/) add a :helpDeps explanation to how the dependency system works (palantir
plugin, lockfile) and how to retrieve structured information about current
dependencies of a given module (in a tree-like output).
* (/) jar checksums, jar checksum computation and validation. This should be
done without intermediate folders (directly on dependency sets).
* (/) verify min. JVM version and exact gradle version on build startup to
minimize odd build side-effects
* (/) Repro-line for failed tests/ runs.
* (/) add a top-level README note about building with gradle (and the required
JVM).
* (/) add an equivalent of 'validate-source-patterns'
(check-source-patterns.groovy) to precommit.
* (/) add an equivalent of 'rat-sources' to precommit.
* (/) add an equivalent of 'check-example-lucene-match-version' (solr only) to
precommit.
* (/) javadoc compilation
Hard-to-implement stuff already investigated:
* (/) (done) -*Printing console output of failed tests.* There doesn't seem
to be any way to do this in a reasonably efficient way. There are onOutput
listeners but they're slow to operate and solr tests emit *tons* of output so
it's an overkill.-
* (!) (LUCENE-9120) *Tests working with security-debug logs or other JVM-early
log output*. Gradle's test runner works by redirecting Java's stdout/ syserr so
this just won't work. Perhaps we can spin the ant-based test runner for such
corner-cases.
Of lesser importance:
* (/) Add an equivalent of 'documentation-lint" to precommit.
* (/) Do not require files to be committed before running precommit. (staged
files are fine).
* (/) add rendering of javadocs (gradlew javadoc)
* (/) identify and port various "regenerate" tasks from ant builds (javacc,
precompiled automata, etc.)
* (/) Add Solr packaging for docs/* (see TODO in packaging/build.gradle;
currently XSLT...)
* I didn't bother adding Solr dist/test-framework to packaging (who'd use it
from a binary distribution?
* (/) There is some python execution in check-broken-links and
check-missing-javadocs, not sure if it's been ported
* (/) Precommit doesn't catch unused imports
* Attach javadocs to maven publications.
* Add test 'beasting' (rerunning the same suite multiple times). I'm afraid
it'll be difficult to run it sensibly because gradle doesn't offer cwd
separation for the forked test runners.
* if you diff solr packaged distribution against ant-created distribution
there are minor differences in library versions and some JARs are excluded/
moved around. I didn't try to force these as everything seems to work (tests,
etc.) – perhaps these differences should be fixed in the ant build instead.
* Fill in POM details in gradle/defaults-maven.gradle so that they reflect the
previous content better (dependencies aside).
* Add any IDE integration layers that should be added (I use IntelliJ and it
imports the project out of the box, without the need for any special tuning).
*{color:#ff}Note:{color}* this builds on the work done by Mark Miller and
Cao Mạnh Đạt but also applies lessons learned from those two efforts:
* *Do not try to do too many things at once*. If we deviate too far from
master, the branch will be hard to merge.
* *Do everything in baby-steps* and add small, independent build fragments
replacing the old ant infrastructure.
* *Try to engage people to run, test and contribute early*. It can't be a
one-man effort. The more people understand and can contribute to the build, the
more healthy it will be.
was:
This task focuses on providing gradle-based build equivalent for Lucene and
Solr (on master branch). See notes below on why this respin is needed.
The code lives on
[jira] [Commented] (SOLR-14641) PeerSync, remove canHandleVersionRanges check
[ https://issues.apache.org/jira/browse/SOLR-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155252#comment-17155252 ] ASF subversion and git services commented on SOLR-14641: Commit 9562dd887a31102f86f710fd99ea53b019a85fcf in lucene-solr's branch refs/heads/jira/SOLR-14641 from Cao Manh Dat [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=9562dd8 ] SOLR-14641: PeerSync, remove canHandleVersionRanges check > PeerSync, remove canHandleVersionRanges check > - > > Key: SOLR-14641 > URL: https://issues.apache.org/jira/browse/SOLR-14641 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > > SOLR-9207 introduces PeerSync with updates range which committed in 6.2 and > 7.0. To maintain backward compatibility at the time we introduce an endpoint > in RealTimeGetComponent to check whether a node support that feature or not. > It served well its purpose and it should be removed to reduce complexity and > a request-response trip for asking that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[GitHub] [lucene-solr] atris commented on pull request #1662: Harden TestBuildingUpMemoryPressure
atris commented on pull request #1662: URL: https://github.com/apache/lucene-solr/pull/1662#issuecomment-656541978 Minimally invasive change -- should solve the test failure. I beasted this around 150 times with no failures and the counts do seem to match. Unless objections, I plan to merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14641) PeerSync Remove canHandleVersionRanges check
[ https://issues.apache.org/jira/browse/SOLR-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cao Manh Dat updated SOLR-14641: Summary: PeerSync Remove canHandleVersionRanges check (was: Remove canHandleVersionRanges check) > PeerSync Remove canHandleVersionRanges check > > > Key: SOLR-14641 > URL: https://issues.apache.org/jira/browse/SOLR-14641 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > > SOLR-9207 introduces PeerSync with updates range which committed in 6.2 and > 7.0. To maintain backward compatibility at the time we introduce an endpoint > in RealTimeGetComponent to check whether a node support that feature or not. > It served well its purpose and it should be removed to reduce complexity and > a request-response trip for asking that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14641) Remove canHandleVersionRanges check
Cao Manh Dat created SOLR-14641: --- Summary: Remove canHandleVersionRanges check Key: SOLR-14641 URL: https://issues.apache.org/jira/browse/SOLR-14641 Project: Solr Issue Type: Improvement Security Level: Public (Default Security Level. Issues are Public) Reporter: Cao Manh Dat Assignee: Cao Manh Dat SOLR-9207 introduces PeerSync with updates range which committed in 6.2 and 7.0. To maintain backward compatibility at the time we introduce an endpoint in RealTimeGetComponent to check whether a node support that feature or not. It served well its purpose and it should be removed to reduce complexity and a request-response trip for asking that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14641) PeerSync, remove canHandleVersionRanges check
[ https://issues.apache.org/jira/browse/SOLR-14641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Cao Manh Dat updated SOLR-14641: Summary: PeerSync, remove canHandleVersionRanges check (was: PeerSync Remove canHandleVersionRanges check) > PeerSync, remove canHandleVersionRanges check > - > > Key: SOLR-14641 > URL: https://issues.apache.org/jira/browse/SOLR-14641 > Project: Solr > Issue Type: Improvement > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Cao Manh Dat >Assignee: Cao Manh Dat >Priority: Major > > SOLR-9207 introduces PeerSync with updates range which committed in 6.2 and > 7.0. To maintain backward compatibility at the time we introduce an endpoint > in RealTimeGetComponent to check whether a node support that feature or not. > It served well its purpose and it should be removed to reduce complexity and > a request-response trip for asking that. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14404) CoreContainer level custom requesthandlers
[
https://issues.apache.org/jira/browse/SOLR-14404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155176#comment-17155176
]
Noble Paul commented on SOLR-14404:
---
{quote}A user might very well have similar logic in client code; I think the
user shouldn't have to add sleeps nor inspect ZK or similar. The user uploaded
2.0 then it expects to use 2.0 immediately after. I think it should be up to
Solr to figure out how to make this work.
{quote}
I didn't make any changes to the test here. The server code is made more robust
to ensure that it's up to date with ZK
> CoreContainer level custom requesthandlers
> --
>
> Key: SOLR-14404
> URL: https://issues.apache.org/jira/browse/SOLR-14404
> Project: Solr
> Issue Type: New Feature
>Reporter: Noble Paul
>Assignee: Noble Paul
>Priority: Major
> Time Spent: 5.5h
> Remaining Estimate: 0h
>
> caveats:
> * The class should be annotated with {{org.apache.solr.api.EndPoint}}.
> Which means only V2 APIs are supported
> * The path should have prefix {{/api/plugin}}
> add a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "full.ClassName",
> "path-prefix" : "some-path-prefix"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> add a plugin from a package
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "add": {
> "name":"myplugin",
> "class": "pkgName:full.ClassName" ,
> "path-prefix" : "some-path-prefix" ,
> "version: "1.0"
> }
> }' http://localhost:8983/api/cluster/plugins
> {code}
> remove a plugin
> {code:java}
> curl -X POST -H 'Content-type:application/json' --data-binary '
> {
> "remove": "myplugin"
> }' http://localhost:8983/api/cluster/plugins
> {code}
> The configuration will be stored in the {{clusterprops.json}}
> as
> {code:java}
> {
> "plugins" : {
> "myplugin" : {"class": "full.ClassName", "path-prefix" : "some-path-prefix" }
> }
> }
> {code}
> example plugin
> {code:java}
> public class MyPlugin {
> private final CoreContainer coreContainer;
> public MyPlugin(CoreContainer coreContainer) {
> this.coreContainer = coreContainer;
> }
> @EndPoint(path = "/$path-prefix/path1",
> method = METHOD.GET,
> permission = READ)
> public void call(SolrQueryRequest req, SolrQueryResponse rsp){
> rsp.add("myplugin.version", "2.0");
> }
> }
> {code}
> This plugin will be accessible on all nodes at
> {{/api/some-path-prefix/path1}}. It's possible to add more methods at
> different paths. Ensure that all paths start with {{$path-prefix}} because
> that is the prefix in which the plugin is registered with. So
> {{/some-path-prefix/path2}} , {{/some-path-prefix/my/deeply/nested/path}} are
> all valid paths.
> It's possible that the user chooses to register the plugin with a different
> name. In that case , use a template variable as follows in paths
> {{/cluster/some/other/path}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
