[jira] [Commented] (SOLR-15089) Allow backup/restoration to Amazon's S3 blobstore
[ https://issues.apache.org/jira/browse/SOLR-15089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271751#comment-17271751 ] Varun Thacker commented on SOLR-15089: -- This is exciting! Maybe we could start by using an S3 Mock? > Allow backup/restoration to Amazon's S3 blobstore > -- > > Key: SOLR-15089 > URL: https://issues.apache.org/jira/browse/SOLR-15089 > Project: Solr > Issue Type: Sub-task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Jason Gerlowski >Priority: Major > > Solr's BackupRepository interface provides an abstraction around the physical > location/format that backups are stored in. This allows plugin writers to > create "repositories" for a variety of storage mediums. It'd be nice if Solr > offered more mediums out of the box though, such as some of the "blobstore" > offerings provided by various cloud providers. > This ticket proposes that a "BackupRepository" implementation for Amazon's > popular 'S3' blobstore, so that Solr users can use it for backups without > needing to write their own code. > Amazon offers a s3 Java client with acceptable licensing, and the required > code is relatively simple. The biggest challenge in supporting this will > likely be procedural - integration testing requires S3 access and S3 access > costs money. We can check with INFRA to see if there is any way to get cloud > credits for an integration test to run in nightly Jenkins runs on the ASF > Jenkins server. Alternatively we can try to stub out the blobstore in some > reliable way. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14614) Add Simplified Aggregation Interface to Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229525#comment-17229525
]
Varun Thacker commented on SOLR-14614:
--
nice!
> Add Simplified Aggregation Interface to Streaming Expression
>
>
> Key: SOLR-14614
> URL: https://issues.apache.org/jira/browse/SOLR-14614
> Project: Solr
> Issue Type: Improvement
> Components: query, query parsers, streaming expressions
>Affects Versions: 7.7.2, 8.4.1
>Reporter: Aroop
>Assignee: Timothy Potter
>Priority: Major
>
> For the Data Analytics use cases the standard use case is:
> # Find a pattern
> # Then Aggregate by certain dimensions
> # Then compute metrics (like count, sum, avg)
> # Sort by a dimension or metric
> # look at top-n
> This functionality has been available over many different interfaces in the
> past on solr, but only streaming expressions have the ability to deliver
> results in a scalable, performant and stable manner for systems that have
> large data to the tune of Big data systems.
> However, one barrier to entry is the query interface, not being simple enough
> in streaming expressions.
> to give an example of how involved the corresponding streaming expression can
> get, to get it to work on large scale systems,{color:#4c9aff} _find top 10
> cities where someone named Alex works with the respective counts_{color}
> {code:java}
> qt=/stream=facet=
> select( top( rollup(sort(by%3D"city+asc",
>+plist(
>
> select(facet(collection1,+q%3D"(*:*+AND+name:alex)",+buckets%3D"city",+bucketSizeLimit%3D"2010",+bucketSorts%3D"count(*)+desc",+count(*)),+city,+count(*)+as+Nj3bXa),
>
> select(facet(collection2,+q%3D"(*:*+AND+name:alex)",+buckets%3D"city",+bucketSizeLimit%3D"2010",+bucketSorts%3D"count(*)+desc",+count(*)),+city,+count(*)+as+Nj3bXa)
> )),
> +over%3D"city",+sum(Nj3bXa)),
> +n%3D"10",+sort%3D"sum(Nj3bXa)+desc"),
> +city,+sum(Nj3bXa)+as+Nj3bXa)
> {code}
> This is a query on an alias with 2 collections behind it representing 2 data
> partitions, which is a requirement of sorts in big data systems. This is one
> of the only ways to get information from Billions of records in a matter of
> seconds. This is awesome in terms of capability and performance.
> But one can see how involved this syntax can be in the current scheme and is
> a barrier to entry for new adopters.
>
> This Jira is to track the work of creating a simplified analytics endpoint
> augmenting streaming expressions.
> a starting proposal is to have the endpoint have these query parameters:
> {code:java}
> /analytics?action=aggregate=*:*=name:alex=city=count=count=desc=10{code}
> This is equivalent to a sql that an analyst would write:
> {code:java}
> select city, count(*) from collection where name = 'alex'
> group by city order by count(*) desc limit 10;{code}
> On the solr side this would get translated to the best possible streaming
> expression using *rollups, top, sort, plist* etc.; but all done transparently
> to the user.
> Heres to making the power of Streaming expressions simpler to use for all.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Updated] (SOLR-14967) CloudSolrClient does not pass _stateVer_ if ModifiableSolrParams isn't used
[
https://issues.apache.org/jira/browse/SOLR-14967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-14967:
-
Description:
If you are using CloudSolrClient and *only* if you are making a request with
ModifiableSolrParams , SolrJ passes __stateVer__ to the server on every
request -
[code|https://github.com/apache/lucene-solr/blob/3730719ff2ba9051278638e34bac91a4f44e9c3e/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BaseCloudSolrClient.java#L920-L927]
This means callers who aren't passing ModifiableSolrParams ( let's say someone
is using {color:#00}MultiMapSolrParams which is totally allowed and correct
) only updates their state.json every 60 seconds and the whole logic of
__stateVer__ comparisons is getting bypassed
{color}
{color:#00}The combination of Solr's handling of __stateVer_
(_https://issues.apache.org/jira/browse/SOLR-14966) and this Jira has a
terrible effect and causes every request to hit ZK ( this caused ZK cpu to fall
of a cliff in our case ){color}
was:
If you are using CloudSolrClient and *only* if you are making a request with
ModifiableSolrParams , SolrJ passes __stateVer__ to the server on every
request -
[code|https://github.com/apache/lucene-solr/blob/3730719ff2ba9051278638e34bac91a4f44e9c3e/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BaseCloudSolrClient.java#L920-L927]
This means callers who aren't passing ModifiableSolrParams ( let's say someone
is using {color:#00}MultiMapSolrParams which is totally allowed and correct
) never end up updating it's state of the collection{color}
{color:#00}The combination of Solr's handling of __stateVer_
(_https://issues.apache.org/jira/browse/SOLR-14966) and this Jira has a
terrible effect and causes every request to hit ZK ( this caused ZK cpu to fall
of a cliff in our case ){color}
> CloudSolrClient does not pass _stateVer_ if ModifiableSolrParams isn't used
> ---
>
> Key: SOLR-14967
> URL: https://issues.apache.org/jira/browse/SOLR-14967
> Project: Solr
> Issue Type: Bug
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Varun Thacker
>Priority: Major
> Attachments: zk-cpu-idle.png
>
>
> If you are using CloudSolrClient and *only* if you are making a request with
> ModifiableSolrParams , SolrJ passes __stateVer__ to the server on every
> request -
> [code|https://github.com/apache/lucene-solr/blob/3730719ff2ba9051278638e34bac91a4f44e9c3e/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BaseCloudSolrClient.java#L920-L927]
>
> This means callers who aren't passing ModifiableSolrParams ( let's say
> someone is using {color:#00}MultiMapSolrParams which is totally allowed
> and correct ) only updates their state.json every 60 seconds and the whole
> logic of __stateVer__ comparisons is getting bypassed
> {color}
>
> {color:#00}The combination of Solr's handling of __stateVer_
> (_https://issues.apache.org/jira/browse/SOLR-14966) and this Jira has a
> terrible effect and causes every request to hit ZK ( this caused ZK cpu to
> fall of a cliff in our case ){color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14967) CloudSolrClient does not pass _stateVer_ if ModifiableSolrParams isn't used
Varun Thacker created SOLR-14967:
Summary: CloudSolrClient does not pass _stateVer_ if
ModifiableSolrParams isn't used
Key: SOLR-14967
URL: https://issues.apache.org/jira/browse/SOLR-14967
Project: Solr
Issue Type: Bug
Security Level: Public (Default Security Level. Issues are Public)
Reporter: Varun Thacker
Attachments: zk-cpu-idle.png
If you are using CloudSolrClient and *only* if you are making a request with
ModifiableSolrParams , SolrJ passes __stateVer__ to the server on every
request -
[code|https://github.com/apache/lucene-solr/blob/3730719ff2ba9051278638e34bac91a4f44e9c3e/solr/solrj/src/java/org/apache/solr/client/solrj/impl/BaseCloudSolrClient.java#L920-L927]
This means callers who aren't passing ModifiableSolrParams ( let's say someone
is using {color:#00}MultiMapSolrParams which is totally allowed and correct
) never end up updating it's state of the collection{color}
{color:#00}The combination of Solr's handling of __stateVer_
(_https://issues.apache.org/jira/browse/SOLR-14966) and this Jira has a
terrible effect and causes every request to hit ZK ( this caused ZK cpu to fall
of a cliff in our case ){color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14966) Solr should use cache collection state to validate _stateVer_ sent by SolrJ
Varun Thacker created SOLR-14966: Summary: Solr should use cache collection state to validate _stateVer_ sent by SolrJ Key: SOLR-14966 URL: https://issues.apache.org/jira/browse/SOLR-14966 Project: Solr Issue Type: Bug Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker Whenever Solr receives a request with the __stateVer__ parameter it's supposed to validate the version the server has vs what the client has. The server was meant to cache the state.json for 2 seconds but it's not doing so. If you follow this [code|https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L928] you'll see _clusterState.getCollectionOrNull(coll)_ resulting it a ZK fetch ( [LazyCollectionRef|https://github.com/apache/lucene-solr/blob/master/solr/solrj/src/java/org/apache/solr/common/cloud/ZkStateReader.java#L656] ) . If the code was doing _clusterState.getCollectionOrNull(coll, true)_ then we would call ZK only after the 2s TTL has expired -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17219975#comment-17219975
]
Varun Thacker commented on SOLR-14354:
--
Whenever we've taken flamegraphs (
[https://github.com/jvm-profiling-tools/async-profiler] with -e wall ) of
production solr clusters, HttpShardHandler has taken significant amount of wall
clock time
!image-2020-10-23-16-45-37-628.png!
I would have been really curious to find out the performance implications of
this change on the cluster. Perhaps in a months timeframe I can try to apply
the patch on top of 8.7 ( we'll first have to upgrade to 8.7 ) and then report
back with some real numbers.
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png,
> image-2020-10-23-16-45-20-034.png, image-2020-10-23-16-45-21-789.png,
> image-2020-10-23-16-45-37-628.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very
[jira] [Updated] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-14354:
-
Attachment: image-2020-10-23-16-45-20-034.png
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png,
> image-2020-10-23-16-45-20-034.png, image-2020-10-23-16-45-21-789.png,
> image-2020-10-23-16-45-37-628.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result result) {
>
[jira] [Updated] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-14354:
-
Attachment: image-2020-10-23-16-45-37-628.png
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png,
> image-2020-10-23-16-45-20-034.png, image-2020-10-23-16-45-21-789.png,
> image-2020-10-23-16-45-37-628.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result result) {
>
[jira] [Updated] (SOLR-14354) HttpShardHandler send requests in async
[
https://issues.apache.org/jira/browse/SOLR-14354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Varun Thacker updated SOLR-14354:
-
Attachment: image-2020-10-23-16-45-21-789.png
> HttpShardHandler send requests in async
> ---
>
> Key: SOLR-14354
> URL: https://issues.apache.org/jira/browse/SOLR-14354
> Project: Solr
> Issue Type: Improvement
>Reporter: Cao Manh Dat
>Assignee: Cao Manh Dat
>Priority: Blocker
> Fix For: master (9.0), 8.7
>
> Attachments: image-2020-03-23-10-04-08-399.png,
> image-2020-03-23-10-09-10-221.png, image-2020-03-23-10-12-00-661.png,
> image-2020-10-23-16-45-20-034.png, image-2020-10-23-16-45-21-789.png,
> image-2020-10-23-16-45-37-628.png
>
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> h2. 1. Current approach (problem) of Solr
> Below is the diagram describe the model on how currently handling a request.
> !image-2020-03-23-10-04-08-399.png!
> The main-thread that handles the search requests, will submit n requests (n
> equals to number of shards) to an executor. So each request will correspond
> to a thread, after sending a request that thread basically do nothing just
> waiting for response from other side. That thread will be swapped out and CPU
> will try to handle another thread (this is called context switch, CPU will
> save the context of the current thread and switch to another one). When some
> data (not all) come back, that thread will be called to parsing these data,
> then it will wait until more data come back. So there will be lots of context
> switching in CPU. That is quite inefficient on using threads.Basically we
> want less threads and most of them must busy all the time, because threads
> are not free as well as context switching. That is the main idea behind
> everything, like executor
> h2. 2. Async call of Jetty HttpClient
> Jetty HttpClient offers async API like this.
> {code:java}
> httpClient.newRequest("http://domain.com/path;)
> // Add request hooks
> .onRequestQueued(request -> { ... })
> .onRequestBegin(request -> { ... })
> // Add response hooks
> .onResponseBegin(response -> { ... })
> .onResponseHeaders(response -> { ... })
> .onResponseContent((response, buffer) -> { ... })
> .send(result -> { ... }); {code}
> Therefore after calling {{send()}} the thread will return immediately without
> any block. Then when the client received the header from other side, it will
> call {{onHeaders()}} listeners. When the client received some {{byte[]}} (not
> all response) from the data it will call {{onContent(buffer)}} listeners.
> When everything finished it will call {{onComplete}} listeners. One main
> thing that will must notice here is all listeners should finish quick, if the
> listener block, all further data of that request won’t be handled until the
> listener finish.
> h2. 3. Solution 1: Sending requests async but spin one thread per response
> Jetty HttpClient already provides several listeners, one of them is
> InputStreamResponseListener. This is how it is get used
> {code:java}
> InputStreamResponseListener listener = new InputStreamResponseListener();
> client.newRequest(...).send(listener);
> // Wait for the response headers to arrive
> Response response = listener.get(5, TimeUnit.SECONDS);
> if (response.getStatus() == 200) {
> // Obtain the input stream on the response content
> try (InputStream input = listener.getInputStream()) {
> // Read the response content
> }
> } {code}
> In this case, there will be 2 thread
> * one thread trying to read the response content from InputStream
> * one thread (this is a short-live task) feeding content to above
> InputStream whenever some byte[] is available. Note that if this thread
> unable to feed data into InputStream, this thread will wait.
> By using this one, the model of HttpShardHandler can be written into
> something like this
> {code:java}
> handler.sendReq(req, (is) -> {
> executor.submit(() ->
> try (is) {
> // Read the content from InputStream
> }
> )
> }) {code}
> The first diagram will be changed into this
> !image-2020-03-23-10-09-10-221.png!
> Notice that although “sending req to shard1” is wide, it won’t take long time
> since sending req is a very quick operation. With this operation, handling
> threads won’t be spin up until first bytes are sent back. Notice that in this
> approach we still have active threads waiting for more data from InputStream
> h2. 4. Solution 2: Buffering data and handle it inside jetty’s thread.
> Jetty have another listener called BufferingResponseListener. This is how it
> is get used
> {code:java}
> client.newRequest(...).send(new BufferingResponseListener() {
> public void onComplete(Result result) {
>
[jira] [Commented] (SOLR-14935) Solr can forward request ( remoteQuery ) even if there are local cores present
[ https://issues.apache.org/jira/browse/SOLR-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17218010#comment-17218010 ] Varun Thacker commented on SOLR-14935: -- I am not using preferLocalShards=true . If you see [https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/servlet/HttpSolrCall.java#L284] , we're just looking at the first collection in the alias list. So depending on the number of collection in your alias the odds will change. > Solr can forward request ( remoteQuery ) even if there are local cores present > -- > > Key: SOLR-14935 > URL: https://issues.apache.org/jira/browse/SOLR-14935 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > Noticed an odd interaction where search queries are being proxied ( treated > as remoteQuery ) even when there are cores from that alias on the querying > node > * Running Solr 7.7.3 and querying via CloudSolrClient > * We use aliases and query with the alias name in CloudSolrClient > * CloudSolrClient creates a list of all replicas from the alias ( after > resolving it ) and then shuffles the list picking one Solr node to query > against > * When Solr receives the request, it only looks at the first collection from > the alias and tries to find a local core > * Anytime that isn't the case, it makes a remoteQuery , proxying the > request to machineX which has a local core of the first collection from the > alias. > > I think we can make Solr check against all collections in the alias to see if > there is a local node present without performance implications. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14935) Solr can forward request ( remoteQuery ) even if there are local cores present
[ https://issues.apache.org/jira/browse/SOLR-14935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214406#comment-17214406 ] Varun Thacker commented on SOLR-14935: -- https://issues.apache.org/jira/browse/SOLR-14630 - Is a very similar issue that touches the same codepath of CloudSolrClient and HttpSolrCall ( where the core matching happens ) > Solr can forward request ( remoteQuery ) even if there are local cores present > -- > > Key: SOLR-14935 > URL: https://issues.apache.org/jira/browse/SOLR-14935 > Project: Solr > Issue Type: Task > Security Level: Public(Default Security Level. Issues are Public) >Reporter: Varun Thacker >Priority: Major > > Noticed an odd interaction where search queries are being proxied ( treated > as remoteQuery ) even when there are cores from that alias on the querying > node > * Running Solr 7.7.3 and querying via CloudSolrClient > * We use aliases and query with the alias name in CloudSolrClient > * CloudSolrClient creates a list of all replicas from the alias ( after > resolving it ) and then shuffles the list picking one Solr node to query > against > * When Solr receives the request, it only looks at the first collection from > the alias and tries to find a local core > * Anytime that isn't the case, it makes a remoteQuery , proxying the > request to machineX which has a local core of the first collection from the > alias. > > I think we can make Solr check against all collections in the alias to see if > there is a local node present without performance implications. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14935) Solr can forward request ( remoteQuery ) even if there are local cores present
Varun Thacker created SOLR-14935: Summary: Solr can forward request ( remoteQuery ) even if there are local cores present Key: SOLR-14935 URL: https://issues.apache.org/jira/browse/SOLR-14935 Project: Solr Issue Type: Task Security Level: Public (Default Security Level. Issues are Public) Reporter: Varun Thacker Noticed an odd interaction where search queries are being proxied ( treated as remoteQuery ) even when there are cores from that alias on the querying node * Running Solr 7.7.3 and querying via CloudSolrClient * We use aliases and query with the alias name in CloudSolrClient * CloudSolrClient creates a list of all replicas from the alias ( after resolving it ) and then shuffles the list picking one Solr node to query against * When Solr receives the request, it only looks at the first collection from the alias and tries to find a local core * Anytime that isn't the case, it makes a remoteQuery , proxying the request to machineX which has a local core of the first collection from the alias. I think we can make Solr check against all collections in the alias to see if there is a local node present without performance implications. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9564) Format code automatically and enforce it
[ https://issues.apache.org/jira/browse/LUCENE-9564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17208840#comment-17208840 ] Varun Thacker commented on LUCENE-9564: --- +1 to consistent and enforced formatting. The one downside is git blame will now show the commit that reformatted the code in all these files. > Format code automatically and enforce it > > > Key: LUCENE-9564 > URL: https://issues.apache.org/jira/browse/LUCENE-9564 > Project: Lucene - Core > Issue Type: Task >Reporter: Dawid Weiss >Assignee: Dawid Weiss >Priority: Trivial > Time Spent: 10m > Remaining Estimate: 0h > > This is a trivial change but a bold move. And I'm sure it's not for everyone. > I started using google java format [1] in my projects a while ago and have > never looked back since. It is an oracle-style formatter (doesn't allow > customizations or deviations from the defined 'ideal') - this takes some > getting used to - but it also eliminates *all* the potential differences > between IDEs, configs, etc. And the formatted code typically looks much > better than hand-edited one. It is also verifiable on precommit (so you can't > commit code that deviates from what you'd get from automated formatting > output). > The biggest benefit I see is that refactorings become such a joy and keep the > code neat, everywhere. Before you commit you just reformat everything > automatically, no matter how much you messed it up. > This isn't a change for everyone. I myself love hand-edited, neat code... but > the reality is that with IDE support for automated code changes and so many > people with different styles working on the same codebase keeping it neat is > a big pain. > Checkstyle and other tools are fine for ensuring certain rules but they don't > take the burden of formatting off your shoulders. This tool does. > Like I said - I had *great* reservations about using it at the beginning but > over time got so used to it that I almost can't live without it now. It's > like magic - you play with the code in any way you like, then run formatting > and it's nice and neat. > The downside is that automated formatting does imply potential merge problems > in backward patches (or any currently existing branches). > Like I said, it is a bold move. Just throwing this for your consideration. > I've added a PR that adds spotless but it's not ready; some files would have > to be excluded as they currently violate header rules. > A more interesting thing is here where the current code is automatically > reformatted, for eyeballing only. > https://github.com/dweiss/lucene-solr/commit/80e8f14ca61a13781bc812967a9e38cdbe656041 > [1] https://google.github.io/styleguide/javaguide.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (SOLR-14909) Add replica is very slow on a large cluster
Varun Thacker created SOLR-14909:
Summary: Add replica is very slow on a large cluster
Key: SOLR-14909
URL: https://issues.apache.org/jira/browse/SOLR-14909
Project: Solr
Issue Type: Task
Security Level: Public (Default Security Level. Issues are Public)
Affects Versions: 7.6
Reporter: Varun Thacker
We create ~100 collections every day for new incoming data
We first issue a create-collection request for all the collections (4 shards
and createNodeSet=empty). This would create collections with no replicas
We then issue async add-replica calls for all the shards creating 1 replica
each. 100 collection X 4 shards = 400 add-replica calls. All the add replica
calls pass the node parameter telling Solr where the replica should be created
The cluster has 190 nodes currently and when we upgraded to Solr 7.7.3 we
noticed that the add replicas took 2 hours and 45 mins to complete! Clearly
something was wrong as the same cluster previously running Solr 7.3.1 was
taking a few mins only.
A thread dump of the overseer showed a 100 threads stuck here ( Why 100? That's
the Solr default thread pool size set by MAX_PARALLEL_TASKS in
OverseerTaskProcessor )
{code:java}
"OverseerThreadFactory-13-thread-1226-processing-n:10.128.18.69:8983_solr"
#11163 prio=5 os_prio=0 cpu=0.69ms elapsed=987.97s tid=0x7f01f8051000
nid=0xd7a waiting for monitor entry [0x7f01c1121000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Object.wait(java.base@11.0.5/Native Method)
- waiting on
at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper$SessionRef.get(PolicyHelper.java:449)
- waiting to re-lock in wait() <0x0007259e6a98> (a java.lang.Object)
at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getSession(PolicyHelper.java:493)
at
org.apache.solr.client.solrj.cloud.autoscaling.PolicyHelper.getReplicaLocations(PolicyHelper.java:121)
at
org.apache.solr.cloud.api.collections.Assign.getPositionsUsingPolicy(Assign.java:382)
at
org.apache.solr.cloud.api.collections.Assign$PolicyBasedAssignStrategy.assign(Assign.java:630)
at
org.apache.solr.cloud.api.collections.Assign.getNodesForNewReplicas(Assign.java:368)
at
org.apache.solr.cloud.api.collections.AddReplicaCmd.buildReplicaPositions(AddReplicaCmd.java:360)
at
org.apache.solr.cloud.api.collections.AddReplicaCmd.addReplica(AddReplicaCmd.java:146)
at
org.apache.solr.cloud.api.collections.AddReplicaCmd.call(AddReplicaCmd.java:91)
at
org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.processMessage(OverseerCollectionMessageHandler.java:294)
{code}
It's strange because each add-replica API call would create a single replica
and specify which node is must be created on.
Assign.getNodesForNewReplicas is where the slowdown was and we noticed
SKIP_NODE_ASSIGNMENT flag (
https://github.com/apache/lucene-solr/commit/17cb1b17172926d0d9aed3dfd3b9adb90cf65e0f#diff-ee29887eff6e474e58fcf3c02077f179R355
) that the overseer reads could have skipped the method from being called.
So we started passing SKIP_NODE_ASSIGNMENT=true and still no luck! The replicas
took just as long to create. It turned out that the Collections Handler wasn't
passing the SKIP_NODE_ASSIGNMENT parameter to the overseer.
The add replica call only passes a specific set of params to the overseer
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/7.7.3/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L823
. We changed this to also pass SKIP_NODE_ASSIGNMENT.
Now when we try to create the replicas it takes 4 minutes approximately vs 2
hours 45 mins that it was taking previosuly.
Only master respects that param to the overseer (
https://github.com/apache/lucene-solr/blob/master/solr/core/src/java/org/apache/solr/handler/admin/CollectionsHandler.java#L938
) . However it doesn't matter in master because the autoscaling framework is
gone ( https://github.com/apache/lucene-solr/commit/cc0c111/ )
I believe this will be seen in all versions since Solr 7.6 (
https://issues.apache.org/jira/browse/SOLR-12739 ) through every 8.x release
Lastly, I manually tried to add a replica with and without the flag. Without
the flag it took 20 second and with the flag 2 seconds.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-14846) Avoid use of Optional.ofNullable.orElse in the same line
[
https://issues.apache.org/jira/browse/SOLR-14846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17192418#comment-17192418
]
Varun Thacker commented on SOLR-14846:
--
Thanks for linking the video! I think the changes makes sense!
> Avoid use of Optional.ofNullable.orElse in the same line
>
>
> Key: SOLR-14846
> URL: https://issues.apache.org/jira/browse/SOLR-14846
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Mike Drob
>Assignee: Mike Drob
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Optional was meant to be used in public APIs to convey to the caller that
> they really need to null-check, not as an internal class to avoid doing our
> own null checks. There are better methods in {{Objects}} for that if we want
> them.
> See also Brian Goetz at JavaOne - https://youtu.be/MFzlgAuanU0?t=2719
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9497) Integerate Error Prone ( Static Analysis Tool ) during compilation
[ https://issues.apache.org/jira/browse/LUCENE-9497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17190810#comment-17190810 ] Varun Thacker commented on LUCENE-9497: --- > error prone uses regular SuppressWarnings annotation with custom names for > each category. yep! We'll use this when we start enabling warnings to suppress legit uses > Integerate Error Prone ( Static Analysis Tool ) during compilation > -- > > Key: LUCENE-9497 > URL: https://issues.apache.org/jira/browse/LUCENE-9497 > Project: Lucene - Core > Issue Type: Task >Reporter: Varun Thacker >Priority: Minor > Time Spent: 4h 20m > Remaining Estimate: 0h > > Integrate [https://github.com/google/error-prone] during compilation of our > source code to catch mistakes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9497) Integerate Error Prone ( Static Analysis Tool ) during compilation
[
https://issues.apache.org/jira/browse/LUCENE-9497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188858#comment-17188858
]
Varun Thacker commented on LUCENE-9497:
---
Here is the first error ( options.errorprone.disableAllWarnings = true )
{code:java}
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/codecs/blocktree/BlockTreeTermsWriter.java:712:
error: [ArrayToString] Calling toString on an array does not provide useful
information
assert StringHelper.startsWith(term.termBytes, prefix): "term.term="
+ term.termBytes + " prefix=" + prefix;
^
(see https://errorprone.info/bugpattern/ArrayToString)
Did you mean 'assert StringHelper.startsWith(term.termBytes, prefix):
"term.term=" + Arrays.toString(term.termBytes) + " prefix=" + prefix;'?
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/codecs/blocktree/BlockTreeTermsWriter.java:744:
error: [ArrayToString] Calling toString on an array does not provide useful
information
assert StringHelper.startsWith(term.termBytes, prefix):
"term.term=" + term.termBytes + " prefix=" + prefix;
^
(see https://errorprone.info/bugpattern/ArrayToString)
Did you mean 'assert StringHelper.startsWith(term.termBytes, prefix):
"term.term=" + Arrays.toString(term.termBytes) + " prefix=" + prefix;'?
2 errors
{code}
> Integerate Error Prone ( Static Analysis Tool ) during compilation
> --
>
> Key: LUCENE-9497
> URL: https://issues.apache.org/jira/browse/LUCENE-9497
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Varun Thacker
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Integrate [https://github.com/google/error-prone] during compilation of our
> source code to catch mistakes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (LUCENE-9497) Integerate Error Prone ( Static Analysis Tool ) during compilation
[
https://issues.apache.org/jira/browse/LUCENE-9497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188851#comment-17188851
]
Varun Thacker edited comment on LUCENE-9497 at 9/1/20, 11:17 PM:
-
Here is the first warning withing lucene core that this tool found
{code:java}
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/search/DocIdSetIterator.java:130:
warning: [IntLongMath] Expression of type int may overflow before being
assigned to a long
return maxDoc - minDoc;
^
(see https://errorprone.info/bugpattern/IntLongMath)
Did you mean 'return (long) maxDoc - minDoc;'?
error: warnings found and -Werror specified
1 error
1 warning
{code}
EDIT: If I suppress that warning ( since it looks like safe usage ) here is
the next warning
{code:java}
> Task :lucene:core:compileJava FAILED
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/search/Sort.java:180:
warning: [ReferenceEquality] Comparison using reference equality instead of
value equality
if (fields[i] != rewrittenSortFields[i]) {
^
(see https://errorprone.info/bugpattern/ReferenceEquality)
Did you mean 'if (!Objects.equals(fields[i], rewrittenSortFields[i])) {' or
'if (!fields[i].equals(rewrittenSortFields[i])) {'?
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/search/Sort.java:185:
warning: [UnnecessaryParentheses] These grouping parentheses are unnecessary;
it is unlikely the code will be misinterpreted without them
return (changed) ? new Sort(rewrittenSortFields) : this;
^
(see https://errorprone.info/bugpattern/UnnecessaryParentheses)
Did you mean 'return changed ? new Sort(rewrittenSortFields) : this;'?
error: warnings found and -Werror specified
1 error
2 warnings
{code}
was (Author: varunthacker):
Here is the first warning withing lucene core that this tool found
{code:java}
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/search/DocIdSetIterator.java:130:
warning: [IntLongMath] Expression of type int may overflow before being
assigned to a long
return maxDoc - minDoc;
^
(see https://errorprone.info/bugpattern/IntLongMath)
Did you mean 'return (long) maxDoc - minDoc;'?
error: warnings found and -Werror specified
1 error
1 warning
{code}
> Integerate Error Prone ( Static Analysis Tool ) during compilation
> --
>
> Key: LUCENE-9497
> URL: https://issues.apache.org/jira/browse/LUCENE-9497
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Varun Thacker
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Integrate [https://github.com/google/error-prone] during compilation of our
> source code to catch mistakes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Created] (LUCENE-9497) Integerate Error Prone ( Static Analysis Tool ) during compilation
Varun Thacker created LUCENE-9497: - Summary: Integerate Error Prone ( Static Analysis Tool ) during compilation Key: LUCENE-9497 URL: https://issues.apache.org/jira/browse/LUCENE-9497 Project: Lucene - Core Issue Type: Task Reporter: Varun Thacker Integrate [https://github.com/google/error-prone] during compilation of our source code to catch mistakes -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9497) Integerate Error Prone ( Static Analysis Tool ) during compilation
[
https://issues.apache.org/jira/browse/LUCENE-9497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17188851#comment-17188851
]
Varun Thacker commented on LUCENE-9497:
---
Here is the first warning withing lucene core that this tool found
{code:java}
/Users/vthacker/apache-work/lucene-solr/lucene/core/src/java/org/apache/lucene/search/DocIdSetIterator.java:130:
warning: [IntLongMath] Expression of type int may overflow before being
assigned to a long
return maxDoc - minDoc;
^
(see https://errorprone.info/bugpattern/IntLongMath)
Did you mean 'return (long) maxDoc - minDoc;'?
error: warnings found and -Werror specified
1 error
1 warning
{code}
> Integerate Error Prone ( Static Analysis Tool ) during compilation
> --
>
> Key: LUCENE-9497
> URL: https://issues.apache.org/jira/browse/LUCENE-9497
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Varun Thacker
>Priority: Minor
>
> Integrate [https://github.com/google/error-prone] during compilation of our
> source code to catch mistakes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13528) Rate limiting in Solr
[ https://issues.apache.org/jira/browse/SOLR-13528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17166137#comment-17166137 ] Varun Thacker commented on SOLR-13528: -- At Slack, we ran into this scenario a few times last year - A node in the cluster would go bad ( maybe a faulty disk ) . This node would end up queuing tens of thousands of outstanding requests because Jetty would keep accepting requests. This then caused issues in our proxy layer ( which is written in Java and uses SolrJ ) by eating up all the threads . Why I mention this is I read this comment in the code - "If available, acquire slot and proceed -- else asynchronously queue the request." - So maybe having an upper bound to the wait queue could be useful? Or is `queryRequestExpirationTimeInMS` for that? We wrote a super simple SearchHandler to implement this. This might not be applicable in a general sense since we were in control of the backend and the solr cluster which helped us simplify out design. the backend would pass a "max_searches_per_node" that would get dynamically set based on the collection / cluster we were running. the search handler would check if it's a top level request and if that nodes concurrent searches is above the threashold or not. If the request is over the limit we would throw a SolrException and the backend could take appropriate steps Two things that we learnt while developing this was * If you throw an exception for rate limit , make sure that error code isn't retried by SolrJ * Preferring a SearchHandler over a QueryComponent - If you write a QueryComponent it's impossible to decrement the counter if the request fails in say the HighlightComponent > Rate limiting in Solr > - > > Key: SOLR-13528 > URL: https://issues.apache.org/jira/browse/SOLR-13528 > Project: Solr > Issue Type: New Feature >Reporter: Anshum Gupta >Assignee: Atri Sharma >Priority: Major > Time Spent: 2h > Remaining Estimate: 0h > > In relation to SOLR-13527, Solr also needs a way to throttle update and > search requests based on usage metrics. This is the umbrella JIRA for both > update and search rate limiting. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165144#comment-17165144
]
Varun Thacker edited comment on SOLR-12845 at 7/26/20, 1:27 AM:
Hi Houston,
Can you post the results of the test you ran with Ishan's script ? Like AB
mentioned on Slack, why was master faster below 80 collections . Wondering if
you also got similar results.
> Have we agreed to revert this commit on 8_6 and 8x?
AB also mentioned that we should remove these rules. I agree with that given
the fact that the autoscaling framework is going away ( SOLR-14656 )
was (Author: varunthacker):
Hi Houston,
Can you post the results of the test you ran with Ishan's script ? Like AB
mentioned on Slack, why was master faster below 80 collections . Wondering if
you also got similar results.
> Have we agreed to revert this commit on 8_6 and 8x?
AB also mentioned that we should remove these rules. I agree with that given
the fact that the autoscaling framework is going away ( SOLR-14656 )
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.6
>
> Attachments: SOLR-12845.patch, SOLR-12845.patch, Screenshot from
> 2020-07-18 21-07-34.png
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-12845) Add a default cluster policy
[
https://issues.apache.org/jira/browse/SOLR-12845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17165144#comment-17165144
]
Varun Thacker commented on SOLR-12845:
--
Hi Houston,
Can you post the results of the test you ran with Ishan's script ? Like AB
mentioned on Slack, why was master faster below 80 collections . Wondering if
you also got similar results.
> Have we agreed to revert this commit on 8_6 and 8x?
AB also mentioned that we should remove these rules. I agree with that given
the fact that the autoscaling framework is going away ( SOLR-14656 )
> Add a default cluster policy
>
>
> Key: SOLR-12845
> URL: https://issues.apache.org/jira/browse/SOLR-12845
> Project: Solr
> Issue Type: Improvement
> Components: AutoScaling
>Reporter: Shalin Shekhar Mangar
>Assignee: Andrzej Bialecki
>Priority: Major
> Fix For: 8.6
>
> Attachments: SOLR-12845.patch, SOLR-12845.patch, Screenshot from
> 2020-07-18 21-07-34.png
>
>
> [~varunthacker] commented on SOLR-12739:
> bq. We should also ship with some default policies - "Don't allow more than
> one replica of a shard on the same JVM" , "Distribute cores across the
> cluster evenly" , "Distribute replicas per collection across the nodes"
> This issue is about adding these defaults. I propose the following as default
> cluster policy:
> {code}
> # Each shard cannot have more than one replica on the same node if possible
> {"replica": "<2", "shard": "#EACH", "node": "#ANY", "strict":false}
> # Each collections replicas should be equally distributed amongst nodes
> {"replica": "#EQUAL", "node": "#ANY", "strict":false}
> # All cores should be equally distributed amongst nodes
> {"cores": "#EQUAL", "node": "#ANY", "strict":false}
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API
[
https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17138707#comment-17138707
]
Varun Thacker commented on LUCENE-9322:
---
JDK
{code:java}
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_242-b08)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.242-b08, mixed mode)
{code}
This is my first time trying out JMH. I took the encoding approach we used in
VectorField vs the encoding approach taken by DenseVectorField ( in SOLR-14397
) and compared them
The VectorField approach to encode is much faster than using Base64 encoding
{code:java}
@Benchmark
public void testVectorFieldEncoding() {
float[] vector = new float[512];
for (int i=0; i<512; i++) {
vector[i] = i + i/1000f;
}
for (int i=0; i<10_000; i++) {
ByteBuffer buffer = ByteBuffer.allocate(Float.BYTES * vector.length);
buffer.asFloatBuffer().put(vector);
buffer.array();
}
}
{code}
JMH output
{code:java}
Result: 123.116 ±(99.9%) 2.671 ops/s [Average]
Statistics: (min, avg, max) = (95.557, 123.116, 143.097), stdev = 11.310
Confidence interval (99.9%): [120.445, 125.787]
# Run complete. Total time: 00:08:07
Benchmark Mode Samples Score Score error Units
o.e.MyBenchmark.testVectorFieldEncoding thrpt 200 123.116 2.671
ops/s
{code}
{code:java}
@Benchmark
public void testBase64Encoding() {
float[] vector = new float[512];
for (int i=0; i<512; i++) {
vector[i] = i + i/1000f;
}
for (int i=0; i<10_000; i++) {
ByteBuffer buffer = ByteBuffer.allocate(Float.BYTES * vector.length);
for (float value : vector) {
buffer.putFloat(value);
}
buffer.rewind();
java.util.Base64.getEncoder().encode(buffer).array();
}
}
{code}
JMH output
{code:java}
Result: 35.069 ±(99.9%) 0.745 ops/s [Average]
Statistics: (min, avg, max) = (25.792, 35.069, 41.335), stdev = 3.154
Confidence interval (99.9%): [34.324, 35.814]
# Run complete. Total time: 00:08:06
Benchmark Mode Samples Score Score error Units
o.e.MyBenchmark.testBase64Encoding thrpt 200 35.069 0.745 ops/s
{code}
> Discussing a unified vectors format API
> ---
>
> Key: LUCENE-9322
> URL: https://issues.apache.org/jira/browse/LUCENE-9322
> Project: Lucene - Core
> Issue Type: New Feature
>Reporter: Julie Tibshirani
>Priority: Major
>
> Two different approximate nearest neighbor approaches are currently being
> developed, one based on HNSW ([#LUCENE-9004]) and another based on coarse
> quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to
> handle vectors. In LUCENE-9136 we discussed the possibility of a unified API
> that could support both approaches. The two ANN strategies give different
> trade-offs in terms of speed, memory, and complexity, and it’s likely that
> we’ll want to support both. Vector search is also an active research area,
> and it would be great to be able to prototype and incorporate new approaches
> without introducing more formats.
> To me it seems like a good time to begin discussing a unified API. The
> prototype for coarse quantization
> ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit
> soon (this depends on everyone's feedback of course). The approach is simple
> and shows solid search performance, as seen
> [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326].
> I think this API discussion is an important step in moving that
> implementation forward.
> The goals of the API would be
> # Support for storing and retrieving individual float vectors.
> # Support for approximate nearest neighbor search -- given a query vector,
> return the indexed vectors that are closest to it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API
[ https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17137909#comment-17137909 ] Varun Thacker commented on LUCENE-9322: --- I've taken only VectorField parts from your PR in [https://github.com/apache/lucene-solr/pull/1584] . This was mostly me trying to to see if it makes sense to break out the work. > Discussing a unified vectors format API > --- > > Key: LUCENE-9322 > URL: https://issues.apache.org/jira/browse/LUCENE-9322 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Julie Tibshirani >Priority: Major > > Two different approximate nearest neighbor approaches are currently being > developed, one based on HNSW ([#LUCENE-9004]) and another based on coarse > quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to > handle vectors. In LUCENE-9136 we discussed the possibility of a unified API > that could support both approaches. The two ANN strategies give different > trade-offs in terms of speed, memory, and complexity, and it’s likely that > we’ll want to support both. Vector search is also an active research area, > and it would be great to be able to prototype and incorporate new approaches > without introducing more formats. > To me it seems like a good time to begin discussing a unified API. The > prototype for coarse quantization > ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit > soon (this depends on everyone's feedback of course). The approach is simple > and shows solid search performance, as seen > [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326]. > I think this API discussion is an important step in moving that > implementation forward. > The goals of the API would be > # Support for storing and retrieving individual float vectors. > # Support for approximate nearest neighbor search -- given a query vector, > return the indexed vectors that are closest to it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9322) Discussing a unified vectors format API
[ https://issues.apache.org/jira/browse/LUCENE-9322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17136805#comment-17136805 ] Varun Thacker commented on LUCENE-9322: --- Hello [~jtibshirani] ! Thanks for tackling this > Support for storing and retrieving individual float vectors. How would we feel to break this part and commit it separately ? I believe this is adding the VectorField field part ? The PR on SOLR-14397 also added a DenseVectorField ( Solr field ) so maybe we could reuse VectorField ( although there is some nuance since DenseVectorField currently supports string and vector encoding and a code comment saying bfloat16 as well ) > Discussing a unified vectors format API > --- > > Key: LUCENE-9322 > URL: https://issues.apache.org/jira/browse/LUCENE-9322 > Project: Lucene - Core > Issue Type: New Feature >Reporter: Julie Tibshirani >Priority: Major > > Two different approximate nearest neighbor approaches are currently being > developed, one based on HNSW ([#LUCENE-9004]) and another based on coarse > quantization ([#LUCENE-9136]). Each prototype proposes to add a new format to > handle vectors. In LUCENE-9136 we discussed the possibility of a unified API > that could support both approaches. The two ANN strategies give different > trade-offs in terms of speed, memory, and complexity, and it’s likely that > we’ll want to support both. Vector search is also an active research area, > and it would be great to be able to prototype and incorporate new approaches > without introducing more formats. > To me it seems like a good time to begin discussing a unified API. The > prototype for coarse quantization > ([https://github.com/apache/lucene-solr/pull/1314]) could be ready to commit > soon (this depends on everyone's feedback of course). The approach is simple > and shows solid search performance, as seen > [here|https://github.com/apache/lucene-solr/pull/1314#issuecomment-608645326]. > I think this API discussion is an important step in moving that > implementation forward. > The goals of the API would be > # Support for storing and retrieving individual float vectors. > # Support for approximate nearest neighbor search -- given a query vector, > return the indexed vectors that are closest to it. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (LUCENE-9086) Benchmark new Graviton2 ARM EC2 instances
[
https://issues.apache.org/jira/browse/LUCENE-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16991874#comment-16991874
]
Varun Thacker commented on LUCENE-9086:
---
When we tested C5.4xlarge vs C4.4xlarge the C5s were able to serve ~31% more
requests per second. These numbers aren't from running lucene however the C5s
might be faster than the C4s ?
> Benchmark new Graviton2 ARM EC2 instances
> -
>
> Key: LUCENE-9086
> URL: https://issues.apache.org/jira/browse/LUCENE-9086
> Project: Lucene - Core
> Issue Type: Task
>Reporter: Michael McCandless
>Assignee: Michael McCandless
>Priority: Major
>
> At [AWS re:Invent 2019|https://reinvent.awsevents.com/] last week, AWS
> announced new [EC2 instances based on the Graviton2 ARM
> processor|https://aws.amazon.com/blogs/aws/coming-soon-graviton2-powered-general-purpose-compute-optimized-memory-optimized-ec2-instances]
> which apparently can be much faster than the original A1 instances, at least
> according to internal benchmarks.
> I've been running Lucene's benchmarks ({{wikimediumall}}, indexing 33.3 M
> docs and running a diverse and repeatable set of search tasks) on these
> instances, comparing a {{c4.8xlarge}} (x86-64) instance against the new
> {{m6g.8xlarge}} (ARM), and I'll summarize the results here.
> Net/net ARM seems to be faster at raw indexing than x86-64, even though
> {{m6g.8xlarge}} has only 32 cores versus 36 cores, but a bit slower at
> merging, while searching seems to be faster for some queries and slower for
> others. I'll try to get the full results posted soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13990) Switch out woodstox-core-asl with aalto-xml and upgrade woodstox stax-2 API
[
https://issues.apache.org/jira/browse/SOLR-13990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16990211#comment-16990211
]
Varun Thacker commented on SOLR-13990:
--
Uwe and Mark had some thoughts on this
[here|https://lucene.472066.n3.nabble.com/Remove-woodstox-core-asl-and-stax2-api-dependencies-from-Solr-td422.html]
> Switch out woodstox-core-asl with aalto-xml and upgrade woodstox stax-2 API
> ---
>
> Key: SOLR-13990
> URL: https://issues.apache.org/jira/browse/SOLR-13990
> Project: Solr
> Issue Type: Improvement
> Security Level: Public(Default Security Level. Issues are Public)
>Reporter: Anshum Gupta
>Assignee: Anshum Gupta
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Switched out woodstox-core-asl with aalto-xml and upgrade woodstax stax-2
> API.
> About aalto-xml:
> Aalto XML processor is an ultra-high performance next generation Stax XML
> processor implementation, implementing both basic Stax API
> ({{javax.xml.stream}}) and Stax2 API extension
> ({{org.codehaus.woodstox.stax2}}). In addition, it also implements SAX2 API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Commented] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967685#comment-16967685
]
Varun Thacker commented on SOLR-13452:
--
I was able to get it running with the security manager by making this change
locally.
{code:java}
~/apache-work/lucene-solr (jira/SOLR-13452_gradle_8) $ git diff
diff --git lucene/tools/junit4/solr-tests.policy
lucene/tools/junit4/solr-tests.policy
index 201add9ed3b..06b08fc91a7 100644
--- lucene/tools/junit4/solr-tests.policy
+++ lucene/tools/junit4/solr-tests.policy
@@ -99,4 +99,7 @@ grant {
// java 8 accessibility requires this perm - should not after 8 I believe
(rrd4j is the root reason we hit an accessibility code path)
permission java.awt.AWTPermission "*";
+ permission java.util.logging.LoggingPermission "control";
+ permission java.util.PropertyPermission "*", "read,write";
+
};
diff --git lucene/tools/junit4/tests.policy lucene/tools/junit4/tests.policy
index 84c67f4452d..861b19495c5 100644
--- lucene/tools/junit4/tests.policy
+++ lucene/tools/junit4/tests.policy
@@ -124,4 +124,7 @@ grant {
// allows LuceneTestCase#runWithRestrictedPermissions to execute with lower
(or no) permission
permission java.security.SecurityPermission "createAccessControlContext";
+
+ permission java.util.logging.LoggingPermission "control";
+ permission java.util.PropertyPermission "*", "read,write";
};
{code}
> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
> -
>
> Key: SOLR-13452
> URL: https://issues.apache.org/jira/browse/SOLR-13452
> Project: Solr
> Issue Type: Improvement
> Components: Build
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: gradle-build.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I took some things from the great work that Dat did in
> [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a
> little further.
>
> When working with gradle in sub modules directly, I recommend
> [https://github.com/dougborg/gdub]
> This gradle branch uses the following plugin for version locking, version
> configuration and version consistency across modules:
> [https://github.com/palantir/gradle-consistent-versions]
>
> https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_8
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967020#comment-16967020
]
Varun Thacker edited comment on SOLR-13452 at 11/4/19 10:43 PM:
What do others get when they try running {{gradlew test}}?
I am running into this
{noformat}
~/apache-work/lucene-solr (jira/SOLR-13452_gradle_8) $ gradlew test
Task :lucene:lucene-backward-codecs:test FAILED
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
at
java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at
java.base/java.security.AccessController.checkPermission(AccessController.java:897)
at
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at
java.logging/java.util.logging.LogManager.checkPermission(LogManager.java:2432)
at java.logging/java.util.logging.LogManager.reset(LogManager.java:1415)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.install(JavaUtilLoggingSystem.java:113)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.startCapture(JavaUtilLoggingSystem.java:78)
at
org.gradle.internal.logging.services.DefaultLoggingManager$StartableLoggingSystem.start(DefaultLoggingManager.java:330)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:81)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:38)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:127)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:70)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:68)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:73)
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
{noformat}
Uncommenting this line from tests.policy and solr-tests.policy didn't help
{noformat}
permission java.util.logging.LoggingPermission "control";{noformat}
And just got kicks I removed this line from the build.gradle file and the tests
started running
{noformat}
ext.testsPolicy = "${rootDir}/lucene/tools/junit4/tests.policy"{noformat}
I'm going to look into it but curious if others ran into similar issues?
was (Author: varunthacker):
What do others get when they try running {{gradle test}}?
I am running into this
{noformat}
~/apache-work/lucene-solr (jira/SOLR-13452_gradle_8) $ gradle test
Task :lucene:lucene-backward-codecs:test FAILED
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
at
java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at
java.base/java.security.AccessController.checkPermission(AccessController.java:897)
at
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at
java.logging/java.util.logging.LogManager.checkPermission(LogManager.java:2432)
at java.logging/java.util.logging.LogManager.reset(LogManager.java:1415)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.install(JavaUtilLoggingSystem.java:113)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.startCapture(JavaUtilLoggingSystem.java:78)
at
org.gradle.internal.logging.services.DefaultLoggingManager$StartableLoggingSystem.start(DefaultLoggingManager.java:330)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:81)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:38)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:127)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:70)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:68)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:73)
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
{noformat}
Uncommenting this line from tests.policy and solr-tests.policy didn't help
{noformat}
permission java.util.logging.LoggingPermission "control";{noformat}
And just got kicks I removed this line from the build.gradle file and the tests
started running
{noformat}
ext.testsPolicy = "${rootDir}/lucene/tools/junit4/tests.policy"{noformat}
I'm going to look
[jira] [Commented] (SOLR-13452) Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
[
https://issues.apache.org/jira/browse/SOLR-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967020#comment-16967020
]
Varun Thacker commented on SOLR-13452:
--
What do others get when they try running {{gradle test}}?
I am running into this
{noformat}
~/apache-work/lucene-solr (jira/SOLR-13452_gradle_8) $ gradle test
Task :lucene:lucene-backward-codecs:test FAILED
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
at
java.base/java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
at
java.base/java.security.AccessController.checkPermission(AccessController.java:897)
at
java.base/java.lang.SecurityManager.checkPermission(SecurityManager.java:322)
at
java.logging/java.util.logging.LogManager.checkPermission(LogManager.java:2432)
at java.logging/java.util.logging.LogManager.reset(LogManager.java:1415)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.install(JavaUtilLoggingSystem.java:113)
at
org.gradle.internal.logging.source.JavaUtilLoggingSystem.startCapture(JavaUtilLoggingSystem.java:78)
at
org.gradle.internal.logging.services.DefaultLoggingManager$StartableLoggingSystem.start(DefaultLoggingManager.java:330)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:81)
at
org.gradle.internal.logging.services.DefaultLoggingManager.start(DefaultLoggingManager.java:38)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:127)
at
org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:70)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:68)
at
worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:73)
java.security.AccessControlException: access denied
("java.util.logging.LoggingPermission" "control")
{noformat}
Uncommenting this line from tests.policy and solr-tests.policy didn't help
{noformat}
permission java.util.logging.LoggingPermission "control";{noformat}
And just got kicks I removed this line from the build.gradle file and the tests
started running
{noformat}
ext.testsPolicy = "${rootDir}/lucene/tools/junit4/tests.policy"{noformat}
I'm going to look into it but curious if others ran into similar issues?
> Update the lucene-solr build from Ivy+Ant+Maven (shadow build) to Gradle.
> -
>
> Key: SOLR-13452
> URL: https://issues.apache.org/jira/browse/SOLR-13452
> Project: Solr
> Issue Type: Improvement
> Components: Build
>Reporter: Mark Miller
>Assignee: Mark Miller
>Priority: Major
> Fix For: master (9.0)
>
> Attachments: gradle-build.pdf
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> I took some things from the great work that Dat did in
> [https://github.com/apache/lucene-solr/tree/jira/gradle] and took the ball a
> little further.
>
> When working with gradle in sub modules directly, I recommend
> [https://github.com/dougborg/gdub]
> This gradle branch uses the following plugin for version locking, version
> configuration and version consistency across modules:
> [https://github.com/palantir/gradle-consistent-versions]
>
> https://github.com/apache/lucene-solr/tree/jira/SOLR-13452_gradle_8
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@lucene.apache.org
For additional commands, e-mail: issues-h...@lucene.apache.org
