Re: Solr Prometheus Exporter failing with "Connection pool shut down" on large cluster
Pull request is here: https://github.com/apache/lucene-solr/pull/1022/ Thanks! Alex Jablonski On Wed, Nov 20, 2019 at 1:36 PM Alex Jablonski wrote: > Hi there! > > My colleague and I have run into an issue that seems to appear when > running the Solr Prometheus exporter in SolrCloud mode against a large (> > 100 node) cluster. The symptoms we're observing are "connection pool shut > down" exceptions in the logs and the inability to collect metrics from more > than 100 nodes in the cluster. > > We think we've traced down the issue to > lucene-solr/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java > . In that class, hostClientCache exists as a cache of HttpSolrClients > (currently having fixed size 100) that, on evicting a client from the > cache, closes the client's connection. The hostClientCache is used in > createHttpSolrClients to return a map of base URLs to HttpSolrClients. > > Given, say, 300 base URLs, createHttpSolrClients will happily add those > base URLs to the cache, and the "get" method on the cache will happily > return the new additions to the cache. But on adding the 101st > HttpSolrClient to the cache, the first HttpSolrClient gets evicted and > closed. This repeats itself until the only open clients we have are to base > URLs 201 through 300; clients for the first 200 base URLs will be returned, > but will already have been closed. When we later use the result of > createHttpSolrClients to collect metrics, expecting valid and open > HttpSolrClients, we fail to connect when using any of those clients that > have already been closed, leading to the "Connection pool shut down" > exception and not collecting metrics from those nodes. > > Our idea for a fix was to change the existing cache to, instead of having > a fixed maximum size, use `expireAfterAccess` with a timeout that's a > multiple of the scrape interval (twice the scrape interval?). We wanted to > confirm a few things: > > 1. Has this issue been reported before, and if so, is there another fix in > progress already? > 2. Does this approach seem desirable? > 3. If so, are there any opinions on what the cache timeout should be > besides just double the scrape interval? > > We'll also open a PR shortly with the changes we're proposing and link > here. Please let me know if any of the above is unclear or incorrect. > > Thanks! > Alex Jablonski > >
Solr Prometheus Exporter failing with "Connection pool shut down" on large cluster
Hi there! My colleague and I have run into an issue that seems to appear when running the Solr Prometheus exporter in SolrCloud mode against a large (> 100 node) cluster. The symptoms we're observing are "connection pool shut down" exceptions in the logs and the inability to collect metrics from more than 100 nodes in the cluster. We think we've traced down the issue to lucene-solr/solr/contrib/prometheus-exporter/src/java/org/apache/solr/prometheus/scraper/SolrCloudScraper.java . In that class, hostClientCache exists as a cache of HttpSolrClients (currently having fixed size 100) that, on evicting a client from the cache, closes the client's connection. The hostClientCache is used in createHttpSolrClients to return a map of base URLs to HttpSolrClients. Given, say, 300 base URLs, createHttpSolrClients will happily add those base URLs to the cache, and the "get" method on the cache will happily return the new additions to the cache. But on adding the 101st HttpSolrClient to the cache, the first HttpSolrClient gets evicted and closed. This repeats itself until the only open clients we have are to base URLs 201 through 300; clients for the first 200 base URLs will be returned, but will already have been closed. When we later use the result of createHttpSolrClients to collect metrics, expecting valid and open HttpSolrClients, we fail to connect when using any of those clients that have already been closed, leading to the "Connection pool shut down" exception and not collecting metrics from those nodes. Our idea for a fix was to change the existing cache to, instead of having a fixed maximum size, use `expireAfterAccess` with a timeout that's a multiple of the scrape interval (twice the scrape interval?). We wanted to confirm a few things: 1. Has this issue been reported before, and if so, is there another fix in progress already? 2. Does this approach seem desirable? 3. If so, are there any opinions on what the cache timeout should be besides just double the scrape interval? We'll also open a PR shortly with the changes we're proposing and link here. Please let me know if any of the above is unclear or incorrect. Thanks! Alex Jablonski
RE: Lucene Upgrade issues.
Hi, If you use it for document filtering queries or to filter deleted documents, then FixedBitSet is the best replacement. The size given in the constructor must be the value of LeafReader#maxDoc(), because that's the maximum document number that can be used and may be marked in the bitset for a given index segment. Uwe - Uwe Schindler Achterdiek 19, D-28357 Bremen https://www.thetaphi.de eMail: u...@thetaphi.de > -Original Message- > From: Jyothsna Bavisetti > Sent: Wednesday, November 20, 2019 2:44 PM > To: dev@lucene.apache.org > Subject: Lucene Upgrade issues. > > Hi All, > > We are upgrading Lucene from 4.6 to 8.3. > Earlier we used Open bit, Now it is deprecated. We have two options in latest > version(FixedBitset or LongBitset). > Could you please let me know which is the best option to use. Is there any > points need to be consider while choosing one of them. If FixedBitSet what > is the minimum size we need to define for it. > > Please share some inputs. > > Thanks, > Jyothsna > > - > To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org > For additional commands, e-mail: dev-h...@lucene.apache.org - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Lucene Upgrade issues.
Hi All, We are upgrading Lucene from 4.6 to 8.3. Earlier we used Open bit, Now it is deprecated. We have two options in latest version(FixedBitset or LongBitset). Could you please let me know which is the best option to use. Is there any points need to be consider while choosing one of them. If FixedBitSet what is the minimum size we need to define for it. Please share some inputs. Thanks, Jyothsna - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Question about Luke
Hello, > Is it different from checkIndex -exorcise option? > (As far as I recently leaned, checkIndex -exorcise will delete unreadable > indices. ) If you mean desktop app Luke, "Repair" is just a wrapper of CheckIndex.exorciseIndex(). There is no difference between doing "Repair" from Luke GUI and calling "CheckIndex -exorcise" from CLI. 2019年11月11日(月) 20:36 Kayak28 : > > Hello, Community: > > I am using Solr7.4.0 currently, and I was testing how Solr actually behaves > when it has a corrupted index. > And I used Luke to fix the broken index from GUI. > I just came up with the following questions. > Is it possible to use the repair index tool from CLI? (in the case, Solr was > on AWS for example.) > Is it different from checkIndex -exorcise option? > (As far as I recently leaned, checkIndex -exorcise will delete unreadable > indices. ) > > If anyone gives me a reply, I would be very thankful. > > Sincerely, > Kaya Ota - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
Re: Solr 8.3 Solrj streaming expressions do not return all field values
No problem, but thanks a lot for the feedback and considerations. I will try to get into the process on how to update the Solr ref guide. I found a lot of things that I could solve to facilitate others to do a solution based on Solr. > Am 20.11.2019 um 13:04 schrieb Joel Bernstein : > > > Ah, I should have noticed that. Yes, there was a change in how string > literals are handled. Quotes are now required for string literals in Math > Expressions. I suspect this change is going to cause problems for other > people as well, but it needed to be done to clarify aspects of the language. > Sorry for the confusion. I'll make a point of updating the documentation to > make sure all examples in the documentation are correct. > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > >> On Tue, Nov 19, 2019 at 5:56 PM Jörn Franke wrote: >> Ok, solved. Solr 8.2 accepted this statement: >> select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", >> qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found) >> >> and return to me the expected results. Note that around Y and N there is no >> ". Solr 8.3 requires the following statement >> select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", >> qt="/export", sort="id asc"),id,if(eq(1,1),"Y","N") as found) >> to return the exact results. I do not know why I did originally not include >> the quotation marks around Y and N, but it seems that 8.2 accepted this and >> 8.3 not. I will update the Jira. >> >> >> >>> On Tue, Nov 19, 2019 at 11:25 PM Jörn Franke wrote: >>> Could it be that Solr 8.3 is more strict on the if statement? >>> the statement if(eq(1,1),Y,N) >>> is supposed to return the character Y (not the field). In Solr 8.2 it >>> returns the character Y, but in Solr 8.3 not. >>> On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke wrote: thanks i will find another installation of Solr, but last time the underlying queries to the export handler were correct. What was wrong was the generated field "found" based on the if statement based NOT on any data in the collectioj. As said, in Solr 8.3 the id fields are returned, but not the "found" field, which is generated by the selected statement: select(search(testcollection,q="test",df="Default",defType="edismax",fl="id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found) returns a tuple id: "blabla" but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 the found field is returned: id: "blabla" found: "Y" To my best knowledge this is a fully generated field => it does not depend on the underlying collection data and the export handler. This was done to exclude any issues with this one. it states simply if 1=1 return Y else N > On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein wrote: > Before moving to a jira let's take a look at the underlying Solr queries > in the log. The Streaming Expressions just creates solr queries, in this > case queries to the /export handler. So when something is not working as > expected we want to strip away the streaming expression and debug the > actual queries that are being run. > > You can find the Solr queries that appear in the log after running the > expressions and then try running them outside of the expression as plain > Solr queries. > > You can also post the Solr queries to this thread and we discuss what the > logs say. > > In these cases the logs always are the way to debug whats going on. > > > Joel Bernstein > http://joelsolr.blogspot.com/ > > >> On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke wrote: >> I created a JIRA for this: >> https://issues.apache.org/jira/browse/SOLR-13894 >> >>> On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke >>> wrote: >>> I have checked now Solr 8.3 server in admin UI. Same issue. >>> >>> Reproduction: >>> select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, >>> qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found) >>> >>> In 8.3 it returns only the id field. >>> In 8.2 it returns id,found field. >>> >>> Since found is generated by select (and not coming from the collection) >>> there must be an issue with select. >>> >>> Any idea why this is happening. >>> >>> Debug logs do not show any error and the expression is correctly >>> received by Solr. >>> >>> Thank you. >>> >>> Best regards >>> >>> > Am 05.11.2019 um 14:59 schrieb Jörn Franke : >>> > >>> > Thanks I will check and come back to you. As far as I remember (but >>> > have to check) the queries generated by Solr were correct >>> > >>> > Just to be clear the same thing works with Solr 8.2 server and Solr >>> > 8.2
Re: Solr 8.3 Solrj streaming expressions do not return all field values
Ah, I should have noticed that. Yes, there was a change in how string literals are handled. Quotes are now required for string literals in Math Expressions. I suspect this change is going to cause problems for other people as well, but it needed to be done to clarify aspects of the language. Sorry for the confusion. I'll make a point of updating the documentation to make sure all examples in the documentation are correct. Joel Bernstein http://joelsolr.blogspot.com/ On Tue, Nov 19, 2019 at 5:56 PM Jörn Franke wrote: > Ok, solved. Solr 8.2 accepted this statement: > select(search(testcollection,q="test",df="Default",defType="edismax",fl= > "id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found) > > and return to me the expected results. Note that around Y and N there is > no ". Solr 8.3 requires the following statement > select(search(testcollection,q="test",df="Default",defType="edismax",fl= > "id", qt="/export", sort="id asc"),id,if(eq(1,1),"Y","N") as found) > to return the exact results. I do not know why I did originally not > include the quotation marks around Y and N, but it seems that 8.2 accepted > this and 8.3 not. I will update the Jira. > > > > On Tue, Nov 19, 2019 at 11:25 PM Jörn Franke wrote: > >> Could it be that Solr 8.3 is more strict on the if statement? >> the statement if(eq(1,1),Y,N) >> is supposed to return the character Y (not the field). In Solr 8.2 it >> returns the character Y, but in Solr 8.3 not. >> >> On Tue, Nov 19, 2019 at 11:21 PM Jörn Franke >> wrote: >> >>> thanks i will find another installation of Solr, but last time the >>> underlying queries to the export handler were correct. What was wrong was >>> the generated field "found" based on the if statement based NOT on any data >>> in the collectioj. As said, in Solr 8.3 the id fields are returned, but >>> not the "found" field, which is generated by the selected statement: >>> select(search(testcollection,q="test",df="Default",defType="edismax",fl= >>> "id", qt="/export", sort="id asc"),id,if(eq(1,1),Y,N) as found) >>> >>> returns a tuple >>> id: "blabla" >>> >>> but not the part generated by the if(eq(1,1),Y,N) as found. In Solr 8.2 >>> the found field is returned: >>> id: "blabla" >>> found: "Y" >>> To my best knowledge this is a fully generated field => it does not >>> depend on the underlying collection data and the export handler. This was >>> done to exclude any issues with this one. >>> it states simply if 1=1 return Y else N >>> >>> >>> On Fri, Nov 8, 2019 at 3:20 PM Joel Bernstein >>> wrote: >>> Before moving to a jira let's take a look at the underlying Solr queries in the log. The Streaming Expressions just creates solr queries, in this case queries to the /export handler. So when something is not working as expected we want to strip away the streaming expression and debug the actual queries that are being run. You can find the Solr queries that appear in the log after running the expressions and then try running them outside of the expression as plain Solr queries. You can also post the Solr queries to this thread and we discuss what the logs say. In these cases the logs always are the way to debug whats going on. Joel Bernstein http://joelsolr.blogspot.com/ On Wed, Nov 6, 2019 at 4:14 PM Jörn Franke wrote: > I created a JIRA for this: > https://issues.apache.org/jira/browse/SOLR-13894 > > On Wed, Nov 6, 2019 at 10:45 AM Jörn Franke > wrote: > >> I have checked now Solr 8.3 server in admin UI. Same issue. >> >> Reproduction: >> select(search(testcollection,q=“test”,df=“Default”,defType=“edismax”,fl=“id”, >> qt=“/export”, sort=“id asc”),id,if(eq(1,1),Y,N) as found) >> >> In 8.3 it returns only the id field. >> In 8.2 it returns id,found field. >> >> Since found is generated by select (and not coming from the >> collection) there must be an issue with select. >> >> Any idea why this is happening. >> >> Debug logs do not show any error and the expression is correctly >> received by Solr. >> >> Thank you. >> >> Best regards >> >> > Am 05.11.2019 um 14:59 schrieb Jörn Franke : >> > >> > Thanks I will check and come back to you. As far as I remember >> (but have to check) the queries generated by Solr were correct >> > >> > Just to be clear the same thing works with Solr 8.2 server and Solr >> 8.2 client. >> > >> > It show the odd behaviour with Solr 8.2 server and Solr 8.3 client. >> > >> >> Am 05.11.2019 um 14:49 schrieb Joel Bernstein > >: >> >> >> >> I'll probably need some more details. One thing that's useful is >> to look at >> >> the logs and see the underlying Solr queries that are generated. >> Then try >> >> those underlying queries against the Solr index and see what comes >> back. If
