Re: Documentation on SolrJ

[Jason Gerlowski]

Hi Thomas,

I recently added a first pass at JSON faceting support to SolrJ.  The
main classes are "JsonQueryRequest" and "DirectJsonQueryRequest" and
live in the package "org.apache.solr.client.solrj.request.json"
(https://github.com/apache/lucene-solr/tree/master/solr/solrj/src/java/org/apache/solr/client/solrj/request/json).
I've also added examples of how to use this code on the "JSON
Faceting" page in the Solr ref guide.  Unfortunately, since this is a
recent addition it hasn't been released yet.  These classes will be in
the next 7x release (if there is one), or in 8.0 when that arrives.
This probably isn't super helpful for you.

Without this code, you have a few options:

1. If the facet requests you'd like to make are relatively
structured/similar, you can subclass QueryRequest and override
getContentWriter().  "ContentWriters" are the abstraction SolrJ is
using to write out the request body.  So you can trivially implement
getContentWriter to wrap a hardcoded string with some templated
variables. If interested, also checkout
"RequestWriter.StringPayloadContentWriter".  This'll be sufficient for
very cookie cutter facet requests, where maybe only a few parameters
change but nothing else.
2. If hardcoding a string JSON body is too inflexible, the JSON
faceting API is "just query params" like everything else.  You can
build your facet request and attach it to the request as a SolrParams
entry.  Doing this wouldn't be the most fun code to write, but it's
always possible.
3. You can copy-paste the unreleased JSON faceting helper classes I
mentioned above into your codebase.  They're not released in SolrJ but
you can still use them by copying them locally and using those copies
until you're able to use a SolrJ that contains these classes.  If you
go this route, please let me or someone else in the community know
your thoughts.  Their being unreleased makes them a bit more of a pain
to use, but it also gives us an opportunity to iterate and improve
them before a release comes and ties us to the existing (maybe awful)
interfaces.

> It would be wonderful if a document of this caliber was provided solely for 
> SolrJ in the form of a tutorial.
We definitely need more "SolrJ Examples" coverage, though I'm not sure
the best way to expose/structure that.  Solr has a *ton* of API
surface area, and SolrJ is responsible for covering all of it.  Even
if I imagine a SolrJ version of the standard "Getting Started"
tutorial which shows users how to create a collection, index docs, do
a query, and do a faceting request...that'd only cover a fraction of
what's out there.  It might be easier to scale our SolrJ examples by
integrating them into the pages we already have for individual APIs
instead.  I'm all for a SolrJ tutorial, or SolrJ Cookbook sort of
thing if you like those ideas better though, and would also volunteer
to help edit or review things in that area.

Sorry, this got a little long.  But hope that helps.

Best,

Jason
On Fri, Nov 30, 2018 at 11:31 AM Cassandra Targett
 wrote:
>
> Support for the JSON Facet API in SolrJ was very recently committed via 
> https://issues.apache.org/jira/browse/SOLR-12965 
> . This missed the cut-off 
> for 7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be 
> able to use the patch there to see if there are gaps or bugs that could be 
> fixed before 7.7 / 8.0.
>
> Jason, who did the work on that issue, also presented on SolrJ at the 
> Activate conference, you may find it interesting:
> https://www.youtube.com/watch?v=ACPUR_GL5zM 
> 
>
> If you do find the time to write some docs, I’d be happy to give you some 
> editing help. Just open a Jira issue when/if you’ve got something and we can 
> go from there.
>
> > On Nov 30, 2018, at 9:53 AM, Thomas L. Redman  wrote:
> >
> > Hi Shawn, thanks for the prompt reply!
> >
> >> On Nov 29, 2018, at 4:55 PM, Shawn Heisey  wrote:
> >>
> >> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
> >>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using 
> >>> SolrJ, and I can find no API for that. I see I can add a pivot field, I 
> >>> guess to a query in general, but that doesn’t seem to work at all, I get 
> >>> an NPE. The documentation on SolrJ is sorely lacking, the documentation I 
> >>> have found is less than a readme. Are there any books that provided a 
> >>> good tretise on SolrJ specifically? Does SolrJ support these more 
> >>> advanced features?
> >>
> >> I don't have any specific details for that use case.
> >
> > Check out page 498 of the PDF, that includes a brief but powerful 
> > discussion of the JSON Facet API. For just one example, I am interested in 
> > faceting a nominal field within a date range bucket. Example: I want to 
> > facet publication_date field into YEAR buckets, and within each YEAR 
> > bucket, facet on author to get the most prolific authors in that year, AND 
> > to also 

Re: solr optimize command

[Christopher Schultz]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA256

Shawn,

On 11/29/18 18:53, Shawn Heisey wrote:
> On 11/29/2018 4:41 PM, Christopher Schultz wrote:
>> When mine returned (with wait=true as a request parameter), I got
>> a JSON response telling me how long it took.
> 
> That's what I would expect.
> 
> If you have to explicitly include parameters like "wait" or 
> "waitSearcher" to make it block until the optimize is done, then in
> my mind, that's a bug.  That should be the default setting.  In the
> 7.5 reference guide, I only see "waitSearcher", and it says the
> default is true.

I didn't test it without that parameter. I used it because it was
suggested to me earlier this week on this list. It may in fact be
optional. I was using Solr 7.4.

- -chris
-BEGIN PGP SIGNATURE-
Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/

iQIzBAEBCAAdFiEEMmKgYcQvxMe7tcJcHPApP6U8pFgFAlwBZjQACgkQHPApP6U8
pFj2ZBAAq741UaizWQkea2dsupyJMUAs+K0A3oHh3Z9QCJqonXdgew620HMmlj2v
iTD1ECZ0OxUy6h4fDKAUFw96FO0/86gsGGMI+BVGZjbBN46oXwpUsNik3gEj3h/E
VjEZ0Nh0qpA783ug2Ezl7zHfeEBd+TRo6tHP1T7S6xp1JFqAs+kB5hxnepipFA/Q
SFssFmdub/0TTDSfxi2taPWxkHVCJO6Atse2HGhiLiRve/ZnV1LabnZnV92OCK6q
YucL3HzrOe23mu1qGJ2uzRM6M8pVkw5QioAUm/ESOFTVv5wqTwMPQ/HGTqO7W/Mp
qU0v3D8+ziKUtCW94UGSEDC5eBOhlr270JWOplYyrxhL/szCCSZ2yVLYaIz6ZXyI
EF5jh1WUsh6w+TrPPN0obUtbN/ZH6SLFzQzocbV6ZhZZL7kqgrAGmw1TVcokR0fC
HhXj0sEukrhRGBaog3+8w21j/ACywb02kTyl21ntpo/+flKHKpitafU2juLHJswD
nb3Q2YAD2bIWX8Ms9QTtozAc+EFVmNw5j2piFprTtWYdbAfqqTS/MxKqZoy/8L49
qiS1lY3eivOGDQufhAhdTO8jTzly5V6Y6xlJ8i0n0oQiPP2FY8yZeCLphdE5Wo/i
jfoauU9WwRGWdq1dwPUe1ZAg9eft2rlvexrVyjh7vjVk92sp17M=
=0Tlc
-END PGP SIGNATURE-

Re: Documentation on SolrJ

[Cassandra Targett]

Support for the JSON Facet API in SolrJ was very recently committed via 
https://issues.apache.org/jira/browse/SOLR-12965 
. This missed the cut-off for 
7.6 but will be included in 7.7 (if there is one) and/or 8.0. You may be able 
to use the patch there to see if there are gaps or bugs that could be fixed 
before 7.7 / 8.0.

Jason, who did the work on that issue, also presented on SolrJ at the Activate 
conference, you may find it interesting:
https://www.youtube.com/watch?v=ACPUR_GL5zM 


If you do find the time to write some docs, I’d be happy to give you some 
editing help. Just open a Jira issue when/if you’ve got something and we can go 
from there.

> On Nov 30, 2018, at 9:53 AM, Thomas L. Redman  wrote:
> 
> Hi Shawn, thanks for the prompt reply!
> 
>> On Nov 29, 2018, at 4:55 PM, Shawn Heisey  wrote:
>> 
>> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
>>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, 
>>> and I can find no API for that. I see I can add a pivot field, I guess to a 
>>> query in general, but that doesn’t seem to work at all, I get an NPE. The 
>>> documentation on SolrJ is sorely lacking, the documentation I have found is 
>>> less than a readme. Are there any books that provided a good tretise on 
>>> SolrJ specifically? Does SolrJ support these more advanced features?
>> 
>> I don't have any specific details for that use case.
> 
> Check out page 498 of the PDF, that includes a brief but powerful discussion 
> of the JSON Facet API. For just one example, I am interested in faceting a 
> nominal field within a date range bucket. Example: I want to facet 
> publication_date field into YEAR buckets, and within each YEAR bucket, facet 
> on author to get the most prolific authors in that year, AND to also facet 
> genre with the same bucket to find out how much scifi, adventure and so on 
> was produced that year. From what I am seeing, beyond pivots(and pivots won’t 
> support this specific use case), I don’t see this capability is supported by 
> the SolrJ API, but this is a hugely powerful feature, and needs to be 
> supported.
> 
> Furthermore, I want to be able to support a vaste range of facets within a 
> single query, perhaps including some collapse and expand, groupings and so on.
> 
>> 
>> If you share the code that gives you NPE, somebody might be able to help you 
>> get it working.
> 
> I haven’t looked in to this enough to drop it in somebody elses' lap at this 
> point, I suspect I am not using the API correctly. And since this won’t allow 
> what I want, I’m not too worried about it.
> 
>> 
>> The best place to find documentation for SolrJ is actually SolrJ itself -- 
>> the javadocs.  Much of that can be accessed pretty easily if you are using 
>> an IDE to do your development.  Here is a link to the top level of the SolrJ 
>> javadocs:
>> 
>> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html 
>> 
> 
> The JavaDocs are limited. I surmise from tracing the code a bit though that I 
> need to rely less on methods provided directly by SolrQuery, and add 
> parameters using methods of the superclasses more frequently. Those 
> superclass methods add simply key value pairs. Still not sure this will allow 
> me the flexibility I need, particularly if the JSON Facet API is not 
> supported.
> 
>> 
>> There's some documentation here, in the official reference guide:
>> 
>> https://lucene.apache.org/solr/guide/7_5/using-solrj.html 
>> 
> 
> This is an excellent document. It would be wonderful if a document of this 
> caliber was provided solely for SolrJ in the form of a tutorial. The existing 
> online tutorial says nothing about how to do anything beyond a simple query. 
> I notice in this document most of the examples of how to issue queries, for 
> example, use curl to issue query. Simply put, this is not a practical 
> approach for the typical user. That being the case, people need to build real 
> UIs around applications that hide the intricacies of the search API. I would 
> rather not build my own API, since SolrJ is already in place, and seems quite 
> powerful. I have been using it for a few years, but really just to do queries.
> 
> I might be interested in contributing to such a document, provided it is 
> sufficiently succinct. I find myself quite busy these days. But I think I 
> would really have to ramp up my understanding of SolrJ to be of any use. Is 
> there any such document in the works, or any interested parties? I am NOT a 
> good writer, I would need somebody to review my work for both accuracy and 
> grammar.
> 
> Also, if the JSON API supported by SolrJ, or is there any plan to support?

Re: Solr on Java 11?

[John Gallagher]

We're interested in this as well.  It is tracked here -
https://issues.apache.org/jira/browse/SOLR-12809

And you can see the test status for different JDKs here:
https://jenkins.thetaphi.de/job/Lucene-Solr-7.x-Linux/

8 and 9 pass completely; 10, 11, 12ea don't

On Fri, Nov 30, 2018 at 10:23 AM Webster Homer <
webster.ho...@milliporesigma.com> wrote:

> My company is planning on upgrading our stack to use Java 11. What version
> of Solr is planned to be supported on Java 11?
> We won't be doing this immediately as several of our key components are
> not yet been ported to 11, but we want to plan for it.
>
> Thanks,
> Webster
>

Re: Documentation on SolrJ

[Thomas L. Redman]

Hi Shawn, thanks for the prompt reply!

> On Nov 29, 2018, at 4:55 PM, Shawn Heisey  wrote:
> 
> On 11/29/2018 2:01 PM, Thomas L. Redman wrote:
>> Hi! I am wanting to do nested facets/Grouping/Expand-Collapse using SolrJ, 
>> and I can find no API for that. I see I can add a pivot field, I guess to a 
>> query in general, but that doesn’t seem to work at all, I get an NPE. The 
>> documentation on SolrJ is sorely lacking, the documentation I have found is 
>> less than a readme. Are there any books that provided a good tretise on 
>> SolrJ specifically? Does SolrJ support these more advanced features?
> 
> I don't have any specific details for that use case.

Check out page 498 of the PDF, that includes a brief but powerful discussion of 
the JSON Facet API. For just one example, I am interested in faceting a nominal 
field within a date range bucket. Example: I want to facet publication_date 
field into YEAR buckets, and within each YEAR bucket, facet on author to get 
the most prolific authors in that year, AND to also facet genre with the same 
bucket to find out how much scifi, adventure and so on was produced that year. 
From what I am seeing, beyond pivots(and pivots won’t support this specific use 
case), I don’t see this capability is supported by the SolrJ API, but this is a 
hugely powerful feature, and needs to be supported.

Furthermore, I want to be able to support a vaste range of facets within a 
single query, perhaps including some collapse and expand, groupings and so on.

> 
> If you share the code that gives you NPE, somebody might be able to help you 
> get it working.

I haven’t looked in to this enough to drop it in somebody elses' lap at this 
point, I suspect I am not using the API correctly. And since this won’t allow 
what I want, I’m not too worried about it.

> 
> The best place to find documentation for SolrJ is actually SolrJ itself -- 
> the javadocs.  Much of that can be accessed pretty easily if you are using an 
> IDE to do your development.  Here is a link to the top level of the SolrJ 
> javadocs:
> 
> https://lucene.apache.org/solr/7_5_0/solr-solrj/index.html 
> 

The JavaDocs are limited. I surmise from tracing the code a bit though that I 
need to rely less on methods provided directly by SolrQuery, and add parameters 
using methods of the superclasses more frequently. Those superclass methods add 
simply key value pairs. Still not sure this will allow me the flexibility I 
need, particularly if the JSON Facet API is not supported.

> 
> There's some documentation here, in the official reference guide:
> 
> https://lucene.apache.org/solr/guide/7_5/using-solrj.html 
> 

This is an excellent document. It would be wonderful if a document of this 
caliber was provided solely for SolrJ in the form of a tutorial. The existing 
online tutorial says nothing about how to do anything beyond a simple query. I 
notice in this document most of the examples of how to issue queries, for 
example, use curl to issue query. Simply put, this is not a practical approach 
for the typical user. That being the case, people need to build real UIs around 
applications that hide the intricacies of the search API. I would rather not 
build my own API, since SolrJ is already in place, and seems quite powerful. I 
have been using it for a few years, but really just to do queries.

I might be interested in contributing to such a document, provided it is 
sufficiently succinct. I find myself quite busy these days. But I think I would 
really have to ramp up my understanding of SolrJ to be of any use. Is there any 
such document in the works, or any interested parties? I am NOT a good writer, 
I would need somebody to review my work for both accuracy and grammar.

Also, if the JSON API supported by SolrJ, or is there any plan to support?

Solr on Java 11?

[Webster Homer]

My company is planning on upgrading our stack to use Java 11. What version of 
Solr is planned to be supported on Java 11?
We won't be doing this immediately as several of our key components are not yet 
been ported to 11, but we want to plan for it.

Thanks,
Webster

Re: Time-Routed Alias Not Distributing Wrongly Placed Docs

[John Nashorn]

Hi Gus, thanks  for writing a detailed answer. I've written some bits between 
quotings from your post.

On 2018/11/30 05:15:10, Gus Heck  wrote: 
> Hi John,
> 
> TRA's really do require that you index via the alias. Internally the code
> is wrapping the Distributed Update Processor with an additional processor
> to handle the time routing when (and only when) the TRA alias is detected.
> If the alias is not used, none of the TRA code runs (by design, for
> performance). TRA's have no capability at all to re-assign docs once they
> are implemented since the process is data driven during update only, with
> no internal maintenance threads (again by design).  It is not even
> supported at this time to update the date on which the document was routed
> via atomic updates for example. One would have to delete and re-index the
> document (in that order, waiting for one to complete!) Adding some sort of
> "fixer thread" is not something that would make much sense, since we don't
> want to ever have the TRA's storing documents in the wrong place to
> begin with.
> 
> TRA's are targeted at systems where new data items arrive regularly, can be
> placed in the right place correctly up front and the timestamp is immutable
> (typical for IOT readings, log or event based types of data for example).
> 
> I think you will probably need to follow up with Lucidworks to get them to
> add a feature to allow TRA's as targets if TRA's still sound like they fit
> your use case. (or pursue another solution without limitations on the
> indexing target)
> 

I know that I'm using TRA out of its designed way, though my scenario would 
perfectly fit for TRA if I were able to use alias name with "hive-solr". I have 
reported the issue to hive-solr devs: 
https://github.com/lucidworks/hive-solr/issues/63

> 
> Frankly, it's a mystery to me how you even got any docs in the October
> collection you list in your question. For anything to have been
> distributed, it would have had to go through the alias. Also, how you have
> more than one collection is a mystery unless you manually inserted a doc at
> some point to cause collection creation perhaps?
> 

Maybe it's the example got you confused, I might have oversummarized it while 
trying to trim. Let me clarify things a little bit: My data ranges from 
2013-01-01 to NOW and continues to grow. I've created a TRA beginning from 
2013-01-01 adding a new collection on a monthly basis. I begun indexing data  
from last to first. Since hive-solr threw NPE when used against TRA name, I was 
sending data to an external table created for the collection of 2013-01-01. 
When the first document was indexed, I saw that all the collections between 
2013-01-01 and 2018-10-01 were created, and the docs were indexed into 
2018-10-01, then 2018-09-01, then 2018-08-01... But after some point, say 
2017-02-01, it stopped this routing and all documents went into 2013-01-01 
collection. 
I didn't manually insert any documents to cause creation of collections.

> 
> It's also worth noting that without the routing and maintenance features
> tied to the alias TRA's give very little benefit, and there are other ways
> of solving this problem with external solutions. Dave, my co-presenter at
> Activate 2018 talks about a couple of other options in the middle section
> of our talk
> https://www.youtube.com/watch?v=RB1-7Y5NQeI=59=PLU6n9Voqu_1HW8-VavVMa9lP8-oF8Oh5t=0s
> 
> 
> The part describing TRA's in detail starts at 14 min and 17 to 23 min
> discusses predecessors and alternatives
> 
> -Gus
> 
> On Tue, Nov 27, 2018 at 12:42 PM John Nashorn  wrote:
> 
> > Hello Everyone,
> > I'm using "hive-solr" from Lucidworks to index my data into Solr (v:7.5,
> > cloud mode). As written in the Solr Manual, TRA expects documents to be
> > indexed using its alias name, and not directly into the collections under
> > it. Unfortunately, hive-solr doesn't allow using TRA names as indexing
> > targets. So what I do is: I index data using the first collection created
> > by TRA and expect Solr to distribute my data into its respective collection
> > under the hood. This works to some extent, but a big portion of data stays
> > in where they were indexed, ie. the first collection of the TRA. For
> > example (approximate numbers):
> >
> > * coll_2018-07-01 => 800.000.000 docs
> > * coll_2018-08-01 => 0 docs
> > * coll_2018-09-01 => 0 docs
> > * coll_2018-10-01 => 150.000.000 docs
> > * coll_2018-11-01 => 0 docs
> >
> > Here, coll_2018-07-01 contains data that should normally be in the other
> > four collections.
> >
> > Is there a way to make TRA scan (somehow intentionally) misplaced data and
> > send them to their correct places?
> >
> 
> 
> -- 
> http://www.the111shift.com
> 

Re: Dataimport UI - shows green even when import fails

[Jan Høydahl]

I have seen the same if the JDBC jar is not found, you cannot tell from the UI, 
you have to go to Solr logs. We should fix this!

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 30. nov. 2018 kl. 00:46 skrev Shawn Heisey :
> 
> I'm looking into a problem where the admin UI dataimport screen has a green 
> status summary bar even though an import has failed.
> 
> Here's a screenshot:
> 
> https://www.dropbox.com/s/94baej11nn75746/dih-green-import-failed.png?dl=0
> 
> What I did to get this:
> Downloaded 7.5.0.
> Extracted the archive.
> On windows 10, in a command prompt at the root of the extracted files:
> bin\solr -e dih
> 
> Then I edited the DIH config for the "db" core, changing the URL to this 
> (just added a "2" near the end):
> 
> url="jdbc:hsqldb:${solr.install.dir}/example/example-DIH/hsqldb/e2x"
> 
> Once that was done, I just clicked the "Execute" button in the dataimport UI 
> for the db core.  The import failed, because the database name in the 
> modified URL doesn't exist.  But the page still shows the status summary in 
> green, with a green check mark.  The screenshot shows "Full Import failed" in 
> the raw status output.  A quick glance at this page will leave a typical user 
> with the incorrect impression that everything is fine with their import.
> 
> I thought I should just go ahead and file a bug, but before I do that, I'd 
> like to know if I should have expected something different here.
> 
> There's been a lot of issues on problems with the fact that the DIH status 
> response is extremely difficult for computers to parse.  It's probably just 
> as hard for the admin UI to parse as it is for most users.  I once wrote some 
> SolrJ code to handle parsing that response.  There was so much code that it 
> needed its own class.
> 
> https://issues.apache.org/jira/browse/SOLR-2728
> https://issues.apache.org/jira/browse/SOLR-2729
> https://issues.apache.org/jira/browse/SOLR-3319
> https://issues.apache.org/jira/browse/SOLR-3689
> https://issues.apache.org/jira/browse/SOLR-4241
> 
> Thanks,
> Shawn
> 

Solr Setup using NRT and PULL replicas

[Daniel Carrasco]

Hello,

We've a cluster consisting in 7 to 10 NRT nodes serving data to a webpage
(products, categories,...), but every time a leader node fails importing
data (connection lost, broken pipe...), the entire cluster goes to
recovering mode and then is not working for about 15-30 minutes. That's a
lot for the webpage so we're trying to reduce that problem to minimal and
we're thinking about new PULL nodes.

We've for now Solr 7.2.1, but we're planning to migrate to Solr 7.5, and
I've read on Solr guide that recommended setups are:

   - All NRT
   - All TLOG
   - Some TLOG with PULL replicas

We're not fully convinced about TLOG replicas because we've read something
about index problems if a node goes down suddenly, or using kill -9 (just
what the init.d script does if takes long to stop/restart), and is the
leader, or about the increase in recovering times respecting to NRT, so
we're thinking about to keep some NRT instead TLOG.

To have two NRT nodes and the rest TLOG is a good setup, or better to think
in TOG nodes?

Thanks!

-- 
_

  Daniel Carrasco Marín
  Ingeniería para la Innovación i2TIC, S.L.
  Tlf:  +34 911 12 32 84 Ext: 223
  www.i2tic.com
_

AW: Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

[Sebastian Riemer]

Dear Jason,

Thank you for your response! I'm happy to tell you that we did resolve our 
issue (by currently downgrading to 6.5 on both solrj-client and server side).

We were formerly executing queries by creating our HttpSolrClient this way (for 
each Query):

SolrClient client = new HttpSolrClient(urlString);

I'm not sure but I guess this way of creating a client is deprecated now? 
Anyhow a colleague changed this to:

SolrClient client = new HttpSolrClient.Builder(urlString).build();

Again, this is done _for each query_ we execute. 

As we discovered by now (like here 
http://lucene.472066.n3.nabble.com/6-6-gt-7-5-SolrJ-seeing-many-quot-Connection-evictor-quot-Threads-td4410488.html)
 this is not the correct way to do it. Instead we should only create one client 
per core and reuse it.

Interestingly enough, with 6.5 the HttpSolrClient.Builder-way of creating a new 
client for every query seems to not create new threads for each query (or it 
closes It automatically or reuses existing I don't know).

Maybe it was a bad idea in the first place, to create a new client for every 
query.

Anyways, thanks a lot for your answer - we'll definitely have to revisit the 
way we create the HttpSolrClient and check for the recommended way of doing so.

All the best,

Sebastian


-Ursprüngliche Nachricht-
Von: Jason Gerlowski [mailto:gerlowsk...@gmail.com] 
Gesendet: Montag, 26. November 2018 17:55
An: solr-user@lucene.apache.org
Betreff: Re: Upgrade 6.2.1 to 7.5.0 - "Connection evictor" Threads not closed

Hey Sebastian,

As for how Solr/SolrJ compatibility is handled, the story for SolrJ looks a lot 
like the story for Solr itself - major version changes can introduce breaking 
changes, so it is best to avoid using SolrJ 6.x with Solr 7.x.  In practice I 
think changes that break Solr/SolrJ compatibility are relatively rare though, 
so it might be possible if your hand is forced.

As for the behavior you described...I think I understand what you're 
describing, but to make sure:  Are the "connection-evictor" threads 
accumulating in your client application, on the Solr server itself, or both?

I suspect you're seeing this in your client code.  If so, it'd really help us 
to help you if you could provide some more details on how you're using SolrJ.  
Can you share a small snippet (JUnit test?) that reproduces the problem?  How 
are you creating the SolrClient you're using to send requests?  Which 
SolrClient implementation(s) are you using?  Are you providing your own 
HttpClient, or letting SolrClient create its own?  It'll be much easier for 
others to help with a little more detail there.

Best,

Jason

On Fri, Nov 23, 2018 at 10:38 AM Sebastian Riemer  wrote:
>
> Hi,
>
> we've recently changed our Solr-Version from 6.2.1 to 7.5.0, and since then, 
> whenever we execute a query on solr, a new thread is being created and never 
> closed.
>
> These threads are all labelled "Connection evictor" and the gather until a 
> critical mass is reached and either the OS cannot create anymore OS threads, 
> or an out of memory error is being produced.
>
> First I thought, that this might have as cause we were using a higher 
> SolrJ-Version than our Solr-Server (by mistakenly forgetting to uprade the 
> server version too):
>
> So we had for SolrJ: 7.4.0
>
> 
> org.apache.solr
> solr-solrj
> 7.4.0
> 
>
> And for Solr-Server:  6.2.1
>
> But now I just installed the newest Solr-Server-Version 7.5.0 and still I see 
> with each Solr-Search performed an additional Thread being created and never 
> released.
>
> When downgrading SolrJ to 6.2.1 I can verify, that no new threads are created 
> when doing a solr search.
>
> What do you think about this? Are there any known pitfalls? Maybe I missed 
> some crucial changes necessary when upgrading to 7.5.0?
>
> What about differing versions in SolrJ and Solr-Server? As far as I recall 
> the docs, one major-version-difference up/down in both ways should be o.k.
>
> Thanks for all your feedback,
>
> Yours sincerely
>
> Sebastian Riemer

Re: solr is using TLS1.0

[Jan Høydahl]

Hmm, perhaps our bin/solr start scripts could set the 
com.ibm.jsse2.overrideDefaultTLS property automatically in case of IBM JVM? Or 
alternatively document this in the SSL section of the reference Guide? Anchal, 
feel free to open a JIRA and submit a patch.

--
Jan Høydahl, search solution architect
Cominvent AS - www.cominvent.com

> 30. nov. 2018 kl. 06:59 skrev Anchal Sharma2 :
> 
> Hi Hendrick 
> 
> This did the trick .Overriding default TLS version for IBM Java enabled TLS 
> 1.2 for solr .
> 
> Thank you Hendrick /Shawn for your help and suggestions.
> 
> Thanks & Regards,
> -
> Anchal Sharma
> 
> 
> Hendrik Haddorp ---22-11-2018 12:53:06---Hi Anchal, the IBM JVM behaves 
> differently in the TLS setup then the Oracle JVM. If
> 
> From: Hendrik Haddorp 
> To: solr-user@lucene.apache.org
> Date: 22-11-2018 12:53
> Subject: Re: solr is using TLS1.0
> 
> 
> 
> 
> Hi Anchal,
> 
> the IBM JVM behaves differently in the TLS setup then the Oracle JVM. If 
> you search for IBM Java TLS 1.2 you find tons of reports of problems 
> with that. In most cases you can get around that using the system 
> property "com.ibm.jsse2.overrideDefaultTLS" as documented here: 
> https://www.ibm.com/support/knowledgecenter/en/SSYKE2_8.0.0/com.ibm.java.security.component.80.doc/security-component/jsse2Docs/matchsslcontext_tls.html
>  
> 
> 
> regards,
> Hendrik
> 
> On 22.11.2018 07:25, Anchal Sharma2 wrote:
> >
> > Hi Shawn ,
> >
> > Thanks for your reply .
> >
> > Here are the details abut java we are using :
> > java version "1.8.0_151"
> > IBM J9 VM (build 2.9, JRE 1.8.0 AIX ppc64-64 Compressed References 
> > 20171102_369060 (JIT enabled, AOT enabled)
> > I have already patched the policy jars .
> >
> > And I tried to comment out the ciphers ,protocol entries in 
> > jetty-ssl.xml ,but it did not work for me .I also tried to use an 
> > "IncludeCipherSuites" entry to include a cipher I wanted to include 
> > ,but it did not work either .I started getting 
> > SSL_ERROR_INTERNAL_ERROR_ALERT and ssl_error_no_cypher_overlap errors 
> > on my console URL.I tried this in solr 7.3.1 version ,so jetty version 
> > must also be relatively new.
> >
> > Do you think java might not be letting me enable TLS1.2?
> >
> > Thanks & Regards,
> > -
> > Anchal Sharma
> >
> >
> > Inactive hide details for Shawn Heisey ---21-11-2018 05:28:50---On 
> > 11/20/2018 3:02 AM, Anchal Sharma2 wrote: > I have enabled Shawn 
> > Heisey ---21-11-2018 05:28:50---On 11/20/2018 3:02 AM, Anchal Sharma2 
> > wrote: > I have enabled SSL for solr using steps mentioned o
> >
> > From: Shawn Heisey 
> > To: solr-user@lucene.apache.org
> > Date: 21-11-2018 05:28
> > Subject: Re: solr is using TLS1.0
> >
> > 
> >
> >
> >
> > On 11/20/2018 3:02 AM, Anchal Sharma2 wrote:
> > > I have enabled  SSL for solr  using steps mentioned over Lucene
> > > website .And though solr console URL is now secure(https) ,it is still
> > > using TLS v1.0.
> > > I have  tried   few things to force SSL to use  TLS1.2 protocol ,but 
> > they
> > > have not worked for me .
> > >
> > > While trying to do same ,I have observed solr itself does not offer any
> > > solr property to specify cipher ,algorithm or TLS version .
> > >
> > > Following things have been tried :
> > > 1.key store /trust store for solr  to enable SSL  with different key
> > > algorithm ,etc combinations for the certificates
> > > 2.different  solr versions for step 1(solr 5.x,6.x,7.x-we are using solr
> > > 5.3 currently)
> > > 3.using java version 1.8 and adding solr certificate in java keystore to
> > > enforce TLS1.2
> >
> > Solr lets Java and Jetty handle TLS.  Solr itself doesn't get involved
> > except to provide information to other software.
> >
> > There are a whole lot of versions of Java 8, and at least three vendors
> > for it.  The big names are Oracle, IBM, and OpenJDK.  What vendor and
> > exact version of Java are you running? What OS is it on?  Do you have
> > the "unlimited JCE" addition installed in your Java and enabled?  If
> > your Java version is new enough, you won't need to mess with JCE.  See
> > this page:
> >
> > https://golb.hplar.ch/2017/10/JCE-policy-changes-in-Java-SE-8u151-and-8u152.html
> >  
> > 
> >
> > Solr 5.3 ships with Jetty 9.2.11, which is considered very outdated by
> > the Jetty project -- released well over three years ago.  From the
> > perspective of the Solr project, version 5.3 is also very old -- two
> > major versions behind what's current, and also released three years ago.
> >
> > Jetty 9.2 is up to 9.2.26.  The current version is Jetty 9.4.14.  The
> > latest