Solr 7.4 and Zookeeper 3.4.12

[Zimmermann, Thomas]

Hi,

I was wondering if there was a reason Solr 7.4 is still recommending ZK 3.4.11 
as the major version in the official changelog vs shipping with 3.4.12 despite 
the known regression in 3.4.11. Are there any known issues with running 7.4 
alongside ZK 3.4.12. We are beginning a major Solr upgrade project (4.10 to 
7.4) and want to stand up the most recent supported versions of both ZK/Solr as 
part of the process.

Thanks,
TZ

SolrJ Kerberos Client API

[Tushar Inamdar]

Hello,

We are looking to move from SolrJ client v5.5.x to the latest v7.4.x.

The documentation on wiring kerberos with the client API here

seems out-of-date. The HttpClientUtil class doesn't have a method
setConfigurer(). Also Krb5HttpClientConfigurer class is missing from the
SolrJ library. This mechanism used to work with v5.5.4, but doesn't work
with any 7.x.

Am I missing something or is the documentation indeed out-of-date?

I am interested in the conventional jaas/keytab based access (not
delegation token).

Thanks,
Tushar.

RE: 7.3 appears to leak

[Markus Jelsma]

Hello Yonik,

If leaking a whole SolrIndexSearcher would cause this problem, then the only 
custom component would be our copy/paste-and-enhance version of the elevator 
component, is the root of all problems. It is a direct copy of the 7.2 source 
where only things like getAnalyzedQuery, the ElevationObj and the loop over the 
map entries is changed.

There are no changes to code related to the searcher. Other component where we 
get a RefCount of searcher is used without issues, we always decrement the 
reference after using it. But those components are not in use in this 
collection.

The source has changed a lot with 7.4 but we still use the old code. I will 
investigate the component thoroughly, even revert to the old 7.2 vanilla 
component for a brief period in production for one machine. It may not be a 
problem if i don't let our load balancer access it directly, so it only serves 
shard queries.

I will get back to this topic tomorrow!

Many thanks,
Markus

 
 
-Original message-
> From:Yonik Seeley 
> Sent: Thursday 28th June 2018 23:30
> To: solr-user@lucene.apache.org
> Subject: Re: 7.3 appears to leak
> 
> > * SortedIntDocSet instances ánd ConcurrentLRUCache$CacheEntry instances are 
> > both leaked on commit;
> 
> If these are actually filterCache entries being leaked, it stands to
> reason that a whole searcher is being leaked somewhere.
> 
> -Yonik
> 

Re: 7.3 appears to leak

[Yonik Seeley]

> * SortedIntDocSet instances ánd ConcurrentLRUCache$CacheEntry instances are 
> both leaked on commit;

If these are actually filterCache entries being leaked, it stands to
reason that a whole searcher is being leaked somewhere.

-Yonik

RE: Custom FieldType not getting populated in the response

[Harsh Verma]

To provide an update here.

I was able to get the plugin to work as expected, but I have to make couple of 
changes - one to solrconfig.xml and another to my query itself.


1.   Disable resultQueryCache in Solr. This fixed the sorting issue for me. 
After result for a query are cached, updates made to redis for updating the 
hits count do not affect the final sorting of the results even though new hits 
value is fetched from redis on each query. I would love to know how can I work 
around it. If values are fetched on each request, I would like Solr to go ahead 
and sort the results again as well.

2.   Instead of just using the field name to get it in the results, I have 
to use field() function. So new query has fl=*,hits:field(hits). Again, not 
ideal and would love to understand what is missing in the implementation that 
is forcing me to use this function format in query.

Here is the working response - 
https://gist.github.com/vermaharsh/d61f5e220f0858ccf9938791da3d3bac

~Harsh

From: Harsh Verma
Sent: Thursday, June 28, 2018 10:42 AM
To: solr-user@lucene.apache.org
Subject: Custom FieldType not getting populated in the response

Hi All,

I am trying to fetch values (docValues to be exact) from external redis during 
query time using a custom written FieldType which I am loading in my solr 
instance. I notice that my custom FieldType is being initialized based on the 
schema when loading the core. I also notice that during query time, my custom 
field type code is called but the results back from solr do not populate the 
custom field even if I explicitly request it. Also, I am asking the results to 
be sorted by the custom field but results are not sorted.

Solr version: 7.3.1
Here is my implementation: 
https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
Here is my  solrconfig.xml: 
https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
Here is my managed-schema: 
https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
Query that I am making: /solr/redis/select?fl=hits=*:*=hits%20asc
Response that I get back: 
https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0

Another query with debug flag: 
solr/redis/select?debugQuery=on=*,hits=*:*=hits%20desc
Response: https://gist.github.com/vermaharsh/f7f74a65a5403ecec9310ceb81cb674c
Sort order should be - 10, 9, 8, 6, 5, 4, 3, 2, 1, 7; because, corresponding 
hits value from redis are - 1, 900, 800, 600, 500, 400, 300, 200, 100, 7.

Can someone help me identify what am I missing?

Thanks,
Harsh

Copy of discussion on #solr irc channel.

 I am trying to write a custom solr FieldType which fetches value from 
external redis instance. I got Solr to load my custom jars and have defined the 
schema with custom fieldType class. But at query time, I cannot get values in 
the response.
[08:38]  I do not see any errors in the logs as well, so cannot tell if 
something failed
[08:38]  here is the code for my custom FieldType - 
https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
[08:39]  this is my solrconfig.xml for my custom configset - 
https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
[08:42]  I basically placed the necessary jars under contrib/redis-field
[08:43]  and my managed-schema - 
https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
[08:43]  I am using solr version 7.3.1
[08:44]  any idea why I am not getting the value for my custom field 
type back?
[08:45] <@elyograg> harsh: I'm not familiar with the API to that level.  FYI, 
this should go in #solr -- this channel is for development of Solr itself.
[08:46] <@elyograg> the field is not marked as stored.  I wonder if that might 
tell Solr that it shouldn't be returned at all.  (I don't know whether setting 
stored=true might require something more in your code, though)
[08:46]  I am using it as docValue, but I can try stored. Though, as you 
mentioned, not sure if that would need more to be implemented in the code
[08:47]  I will try #solr channel for the question as well
[08:47] <@elyograg> ah, I didn't scroll to the right enough to see that part. :)
[08:48] <@elyograg> I wonder if you might need useDocValuesAsStored="true".
[08:49]  I thought that is the default value
[08:51]  for completeness, this is the query that I am using - 
/solr/redis/select?fl=hits=*:*=hits%20asc
[08:52]  and response that I got back - 
https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
[08:53] == dataminion [~le...@c-69-181-118-61.hsd1.ca.comcast.net] has joined 
#solr-dev
[08:53] <@elyograg> this one's probably going to need to go to the mailing 
list.  with all the pastes you've mentioned here.
[08:57] == dataminion [~le...@c-69-181-118-61.hsd1.ca.comcast.net] has quit 
[Ping timeout: 264 seconds]
[08:59] <@elyograg> you may be right about that being default.  The code in 
FieldType.java seems to support that.
[09:00] <@elyograg> if (schemaVersion >= 1.6f) properties |= 

Re: Importance of having the lsof utility on our solr server VMs

[Shawn Heisey]

On 6/28/2018 11:22 AM, THADC wrote:
> Today I am running the ./bin/solr command from new VMs that apparently don't
> have the lsof command installed. I noticed a warning message saying the
> command needs lsof in order to determine if solr is listening on the port I
> am trying to have it run on. But then is says its started anyway.
>
> I did a search on this archive and found thread that said as long as each
> node is started separately, then you should be ok. We will probably always
> be firing up the nodes individually anyway. So, can I safely assume we don't
> need lsof?

If you can be absolutely certain when you ask for Solr to be started
that there is no possibility that something else is already listening on
the Solr port, then you probably don't actually need lsof.

If you do have lsof, and it finds something already listening on the
Solr port, then the script will exit, and it won't even try to start
Solr.  Without lsof, it cannot know whether the port is in use, so it
will start, but if the port is in use, Solr will not start properly.

Thanks,
Shawn

Re: 7.4.0 changes in DocTransformer behaviour

[Shawn Heisey]

On 6/28/2018 8:15 AM, Markus Jelsma wrote:
> Ah, forgot about that thread. But yes, it makes sense and we used it as such 
> in production. But the test was never updated, never failed, until 7.4.0 came 
> around. So it is confusing why the tests didn't fail, and confusing why it 
> does in 7.4.0. Any ideas why now?

On a second look at what Hoss said previously, I think I understand it
now.  Because the name provided when using a transformer is a display
name and not actually a field name, that field is not requested
explicitly from the index, so there's nothing for the transformer to
work on.  But when * is used, the field IS requested, and it works.

So the recommendation was to have the transformer have a parameter to
indicate what source field to use, and override getExtraRequestFields()
to request that field.

RawValueTransformerFactory also shows how to use the display identifier
as the source field name if the parameter indicating the source field is
not present.  This is in the factory's create method.

If you haven't already (and it sounds like maybe you have), I would
recommend you use the same approach as RawValueTransformerFactory, which
would explicitly request the field with the same name as the display if
the parameter for the source field is missing.

I have no idea why your test code would work properly and suddenly break
in 7.4.0.  Is it possible that when you upgraded Solr in the test
environment, that somehow it started loading an old version of the
transformer that doesn't have the changes you made for production back
in 2016?

Thanks,
Shawn

Re: Retrieving json.facet from a search

[Yonik Seeley]

There isn't typed support, but you can use the generic support like so:

.getResponse().get("facets")

-Yonik

On Thu, Jun 28, 2018 at 2:31 PM, Webster Homer  wrote:
> I have a fairly large existing code base for querying Solr. It is
> architected where common code calls solr and returns a solrj QueryResponse
> object.
>
> I'm currently using Solr 7.2 the code interacts with solr using the Solrj
> client api
>
> I have a need that would be very easily met by using the json.facet api.
> The problem is that I don't see how to get the json.facet out of a
> QueryResponse object.
>
> There doesn't seem to be a lot of discussion on line about this.
> Is there a way to get the Json object out of the QueryResponse?
>
> --
>
>
> This message and any attachment are confidential and may be
> privileged or
> otherwise protected from disclosure. If you are not the intended
> recipient,
> you must not copy this message or attachment or disclose the
> contents to
> any other person. If you have received this transmission in error,
> please
> notify the sender immediately and delete the message and any attachment
>
> from your system. Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do
> not accept liability for any omissions or errors in this
> message which may
> arise as a result of E-Mail-transmission or for damages
> resulting from any
> unauthorized changes of the content of this message and
> any attachment thereto.
> Merck KGaA, Darmstadt, Germany and any of its
> subsidiaries do not guarantee
> that this message is free of viruses and does
> not accept liability for any
> damages caused by any virus transmitted
> therewith.
>
>
>
> Click http://www.emdgroup.com/disclaimer
>  to access the
> German, French, Spanish
> and Portuguese versions of this disclaimer.

[ANNOUNCE] Solr Reference Guide for 7.4 released

[Cassandra Targett]

The Lucene PMC is pleased to announce that the Solr Reference Guide for Solr
 7.4 is now available.

This 1,258 page PDF is the definitive guide to using Apache Solr, the
search server built on Apache Lucene.

The PDF Guide can be downloaded from:
https://www.apache.org/dyn/closer.cgi/lucene/solr/ref-guide/apache-solr-ref-
guide-7.4.pdf

It is also available online at https://lucene.apache.org/solr/guide/7_4.

Retrieving json.facet from a search

[Webster Homer]

I have a fairly large existing code base for querying Solr. It is
architected where common code calls solr and returns a solrj QueryResponse
object.

I'm currently using Solr 7.2 the code interacts with solr using the Solrj
client api

I have a need that would be very easily met by using the json.facet api.
The problem is that I don't see how to get the json.facet out of a
QueryResponse object.

There doesn't seem to be a lot of discussion on line about this.
Is there a way to get the Json object out of the QueryResponse?

-- 


This message and any attachment are confidential and may be
privileged or 
otherwise protected from disclosure. If you are not the intended
recipient, 
you must not copy this message or attachment or disclose the
contents to 
any other person. If you have received this transmission in error,
please 
notify the sender immediately and delete the message and any attachment

from your system. Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do
not accept liability for any omissions or errors in this 
message which may
arise as a result of E-Mail-transmission or for damages 
resulting from any
unauthorized changes of the content of this message and 
any attachment thereto.
Merck KGaA, Darmstadt, Germany and any of its 
subsidiaries do not guarantee
that this message is free of viruses and does 
not accept liability for any
damages caused by any virus transmitted 
therewith.



Click http://www.emdgroup.com/disclaimer 
 to access the
German, French, Spanish 
and Portuguese versions of this disclaimer.

Custom FieldType not getting populated in the response

[Harsh Verma]

Hi All,

I am trying to fetch values (docValues to be exact) from external redis during 
query time using a custom written FieldType which I am loading in my solr 
instance. I notice that my custom FieldType is being initialized based on the 
schema when loading the core. I also notice that during query time, my custom 
field type code is called but the results back from solr do not populate the 
custom field even if I explicitly request it. Also, I am asking the results to 
be sorted by the custom field but results are not sorted.

Solr version: 7.3.1
Here is my implementation: 
https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
Here is my  solrconfig.xml: 
https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
Here is my managed-schema: 
https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
Query that I am making: /solr/redis/select?fl=hits=*:*=hits%20asc
Response that I get back: 
https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0

Another query with debug flag: 
solr/redis/select?debugQuery=on=*,hits=*:*=hits%20desc
Response: https://gist.github.com/vermaharsh/f7f74a65a5403ecec9310ceb81cb674c
Sort order should be - 10, 9, 8, 6, 5, 4, 3, 2, 1, 7; because, corresponding 
hits value from redis are - 1, 900, 800, 600, 500, 400, 300, 200, 100, 7.

Can someone help me identify what am I missing?

Thanks,
Harsh

Copy of discussion on #solr irc channel.

 I am trying to write a custom solr FieldType which fetches value from 
external redis instance. I got Solr to load my custom jars and have defined the 
schema with custom fieldType class. But at query time, I cannot get values in 
the response.
[08:38]  I do not see any errors in the logs as well, so cannot tell if 
something failed
[08:38]  here is the code for my custom FieldType - 
https://gist.github.com/vermaharsh/042e1cf07070c6d9b3b6cc7eaaf0b49c
[08:39]  this is my solrconfig.xml for my custom configset - 
https://gist.github.com/vermaharsh/97d7310b242fd7ba3d8d3a3bda209ac3
[08:42]  I basically placed the necessary jars under contrib/redis-field
[08:43]  and my managed-schema - 
https://gist.github.com/vermaharsh/8a89195377802a6bbcdde9215a2fdaf5
[08:43]  I am using solr version 7.3.1
[08:44]  any idea why I am not getting the value for my custom field 
type back?
[08:45] <@elyograg> harsh: I'm not familiar with the API to that level.  FYI, 
this should go in #solr -- this channel is for development of Solr itself.
[08:46] <@elyograg> the field is not marked as stored.  I wonder if that might 
tell Solr that it shouldn't be returned at all.  (I don't know whether setting 
stored=true might require something more in your code, though)
[08:46]  I am using it as docValue, but I can try stored. Though, as you 
mentioned, not sure if that would need more to be implemented in the code
[08:47]  I will try #solr channel for the question as well
[08:47] <@elyograg> ah, I didn't scroll to the right enough to see that part. :)
[08:48] <@elyograg> I wonder if you might need useDocValuesAsStored="true".
[08:49]  I thought that is the default value
[08:51]  for completeness, this is the query that I am using - 
/solr/redis/select?fl=hits=*:*=hits%20asc
[08:52]  and response that I got back - 
https://gist.github.com/vermaharsh/2f63282b10320c4c35a9f85016fe30c0
[08:53] == dataminion [~le...@c-69-181-118-61.hsd1.ca.comcast.net] has joined 
#solr-dev
[08:53] <@elyograg> this one's probably going to need to go to the mailing 
list.  with all the pastes you've mentioned here.
[08:57] == dataminion [~le...@c-69-181-118-61.hsd1.ca.comcast.net] has quit 
[Ping timeout: 264 seconds]
[08:59] <@elyograg> you may be right about that being default.  The code in 
FieldType.java seems to support that.
[09:00] <@elyograg> if (schemaVersion >= 1.6f) properties |= 
USE_DOCVALUES_AS_STORED;
[09:02]  alright, thanks elyograg for looking into it. I will send this 
to mailing list as well.

Importance of having the lsof utility on our solr server VMs

[THADC]

Hello,

Today I am running the ./bin/solr command from new VMs that apparently don't
have the lsof command installed. I noticed a warning message saying the
command needs lsof in order to determine if solr is listening on the port I
am trying to have it run on. But then is says its started anyway.

I did a search on this archive and found thread that said as long as each
node is started separately, then you should be ok. We will probably always
be firing up the nodes individually anyway. So, can I safely assume we don't
need lsof?


Thanks!



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: 7.3 appears to leak

[Markus Jelsma]

Hello again,

Back to this topic, upgrade to 7.4 didn't mysteriously fix the leak our main 
text search collection has as i had so vigorously hoped. Again, it are 
SortedIntDocSet instances that leak consistently on the 15 minute index/commit 
interval.

Some facts:
* problem started after upgrading from 7.2.1 to 7.3.0;
* it occurs only in our main text search collection, all other collections are 
unaffected;
* despite what i said earlier, it is so far unreproducible outside production, 
even when mimicking production as good as we can;
* SortedIntDocSet instances ánd ConcurrentLRUCache$CacheEntry instances are 
both leaked on commit;
* filterCache is enabled using FastLRUCache;
* filter queries are simple field:value using strings, and three filter query 
for time range using [NOW/DAY TO NOW+1DAY/DAY] syntax for 'today', 'last week' 
and 'last month', but rarely used;
* reloading the core manually frees OldGen;
* custom URP's don't cause the problem, disabling them doesn't solve it;
* the collection uses custom extensions for QueryComponent and 
QueryElevationComponent, ExtendedDismaxQParser and MoreLikeThisQParser, a whole 
bunch of TokenFilters, and several DocTransformers and due it being only 
reproducible on production, i really cannot switch these back to Solr/Lucene 
versions;
* useFilterForSortedQuery is/was not defined in schema so it was default 
(true?), SOLR-11769 could be the culprit, i disabled it just now only for the 
node running 7.4.0, rest of collection runs 7.2.1;

The 7.4.0 node with useFilterForSortedQuery=false now seems to be running fine 
for the last three commits. While typing this i may just have been lucky after 
so many hours/days of tediousness. To confirm i will run 7.4.0 on a second node 
in the cluster, but with different values for useFilterForSortedQuery...

I am unlucky after all :( so i'll revert to 7.2.1 again (but why did it 'seem' 
to run fine for three commits?). But we need it fixed and it is clear whatever 
i do, i am not one damn step closer to solving this. So what next? I need the 
list's help to find the leak.

So please, thanks,
Markus
 
-Original message-
> From:Shalin Shekhar Mangar 
> Sent: Friday 27th April 2018 12:11
> To: solr-user@lucene.apache.org
> Subject: Re: 7.3 appears to leak
> 
> Hi Markus,
> 
> Can you give an idea of what your filter queries look like? Any custom
> plugins or things we should be aware of? Simple indexing artificial docs,
> querying and committing doesn't seem to reproduce the issue for me.
> 
> On Thu, Apr 26, 2018 at 10:13 PM, Markus Jelsma 
> wrote:
> 
> > Hello,
> >
> > We just finished upgrading our three separate clusters from 7.2.1 to 7.3,
> > which went fine, except for our main text search collection, it appears to
> > leak memory on commit!
> >
> > After initial upgrade we saw the cluster slowly starting to run out of
> > memory within about an hour and a half. We increased heap in case 7.3 just
> > requires more of it, but the heap consumption graph is still growing on
> > each commit. Heap space cannot be reclaimed by forcing the garbage
> > collector to run, everything just piles up in the OldGen. Running with this
> > slightly larger heap, the first nodes will run out of memory in about two
> > and a half hours after cluster restart.
> >
> > The heap eating cluster is a 2shard/3replica system on separate nodes.
> > Each replica is about 50 GB in size and about 8.5 million documents. On
> > 7.2.1 it ran fine with just a 2 GB heap. With 7.3 and 2.5 GB heap, it will
> > take just a little longer for it to run out of memory.
> >
> > I inspected reports shown by the sampler of VisualVM and spotted one
> > peculiarity, the number of instances of SortedIntDocSet kept growing on
> > each commit by about the same amount as the number of cached filter
> > queries. But this doesn't happen on the logs cluster, SortedIntDocSet
> > instances are neatly collected there. The number of instances also accounts
> > for the number of commits since start up times the cache sizes
> >
> > Our other two clusters don't have this problem, one of them receives very
> > few commits per day, but the other receives data all the time, it logs user
> > interactions so a large amount of data is coming in all the time. I cannot
> > reproduce it locally by indexing data and committing all the time, the peak
> > usage in OldGen stays about the same. But, i can reproduce it locally when
> > i introduce queries, and filter queries while indexing pieces of data and
> > committing it.
> >
> > So, what is the problem? I dug in the CHANGES.txt of both Lucene and Solr,
> > but nothing really caught my attention. Does anyone here have an idea where
> > to look?
> >
> > Many thanks,
> > Markus
> >
> 
> 
> 
> -- 
> Regards,
> Shalin Shekhar Mangar.
> 

Caching of dynamic external file fields

[Zisis T.]

In Solr there's /ExternalFileFieldReloader/ which is responsible for caching
the contents of external files whenever a new searcher is being warmed up. 

It happens that I've defined a dynamic field to be used as an
/ExternalField/ as in 
/* */ 

If you have a look inside the code /ExternalFileFieldReloader/ there's a
loop over the explicit schema fields through /schema.getFields()/ which
means that dynamic external fields do not get cached. 

Of course I did notice that once those files started getting bigger (~50MB)
and searches started timing out. 

Q1 : Is there a specific reason why only explicit fields are cached?
Q2 : I have extended /ExternalFileFieldReloader/ and added the capability to
also cache dynamic external file field contents and it seems to be working
fine. Does anyone think this is useful to make it into Solr codebase?



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: cannot download the config directory with zkcli.bat from embedded zookeeper

[Arturas Mazeika]

Hi Shawn et al,


I think I got it working.

I disabled the firewall, and then I was able to download and upload the
configdirs.

Interestingly enough, after I re-enabled the firewall, I was able to
continue to upload and download the configdirs. As if the problem did not
take place at all. I am highly puzzled. Comments are very welcome.

I'll keep an eye on what's going on and keep you guys informed. Thanks for
the support!!

Cheers,
Arturas







On Thu, Jun 28, 2018 at 6:07 PM, Arturas Mazeika  wrote:

> Hi Shawn, et al,
>
> I haven't thought about ip6. Interesting idea.
>
> - I've tried to use the address 127.0.0.1. It did not help (I've
> un-commented the suppress-output-line in the script, so it is possible to
> see what it is being done) [1]. I also tried all possible combinations of
> the IP addresses that got assigned through the net cards without no success
> [2]. There is some 10sec delay between the start of the command and the
> error.
>
> - I also reset ip6v as you suggested without any success [3].
>
> - ping returns in my case as in yours [4].
>
> - ipconfig /all returns [5].
>
> - Then I disables ipv6 on the network card. No change.
>
> - Then I found
>
> https://superuser.com/questions/586144/disable-ipv6-loopback
> -on-windows-7-64-bit
>
> and did [6]. No change.
>
> I restarted the cluster with -z 127.0.0.1:10999 [6]. No change.
>
> Any comments are very helpful.
>
> Cheers,
> Arturas
>
> References:
>
> [1] F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>f:\
> solr_server\solr-7.2.1\server\scripts\cloud-scripts\zkcli.bat -zkhost
> 127.0.0.1:10999 -cmd downconfig -confdir d:\de_wiki_man_config2 -confname
> de_wiki_man
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>rem @echo off
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM You can
> override pass the following parameters to this script:
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>set JVM=java
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM  Find location
> of this script
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>set
> SDIR=f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>if "\" == "\" set
> SDIR=f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>if defined
> LOG4J_PROPS (set "LOG4J_CONFIG=file:" )  else (set
> "LOG4J_CONFIG=file:f:\solr_server\solr-7.2.1\server\scripts\
> cloud-scripts\log4j.properties" )
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM Settings for
> ZK ACL
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM set
> SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.
> common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider ^
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
> -DzkCredentialsProvider=org.apache.solr.common.cloud.VMParam
> sSingleSetCredentialsDigestZkCredentialsProvider ^
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
> -DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
> -DzkDigestReadonlyUsername=readonly-user -DzkDigestReadonlyPassword=CHA
> NGEME-READONLY-PASSWORD
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM "java"
> -Dlog4j.configuration="file:f:\solr_server\solr-7.2.1\server
> \scripts\cloud-scripts\log4j.properties" ^
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM -classpath
> "f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\.
> .\solr-webapp\webapp\WEB-INF\lib\*;f:\solr_server\solr-7.2.
> 1\server\scripts\cloud-scripts\..\..\lib\ext\*;f:\solr_
> server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\*"
> org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:10999 -cmd downconfig
> -confdir d:\de_wiki_man_config2 -confname de_wiki_man
>
> F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>"java"
> -Dlog4j.configuration="file:f:\solr_server\solr-7.2.1\server
> \scripts\cloud-scripts\log4j.properties" -Djava.net.preferIPv4Stack=true
> -classpath "f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\.
> .\solr-webapp\webapp\WEB-INF\lib\*;f:\solr_server\solr-7.2.
> 1\server\scripts\cloud-scripts\..\..\lib\ext\*;f:\solr_
> server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\*"
> org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:10999 -cmd downconfig
> -confdir d:\de_wiki_man_config2 -confname de_wiki_man
> Exception in thread "main" org.apache.solr.common.SolrException:
> java.io.IOException: Couldn't instantiate org.apache.zookeeper.ClientCnx
> nSocketNIO
> at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClien
> t.java:170)
> at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClien
> t.java:119)
> at org.apache.solr.common.cloud.SolrZkClient.(SolrZkClien
> t.java:109)
> at 

Re: cannot download the config directory with zkcli.bat from embedded zookeeper

[Arturas Mazeika]

Hi Shawn, et al,

I haven't thought about ip6. Interesting idea.

- I've tried to use the address 127.0.0.1. It did not help (I've
un-commented the suppress-output-line in the script, so it is possible to
see what it is being done) [1]. I also tried all possible combinations of
the IP addresses that got assigned through the net cards without no success
[2]. There is some 10sec delay between the start of the command and the
error.

- I also reset ip6v as you suggested without any success [3].

- ping returns in my case as in yours [4].

- ipconfig /all returns [5].

- Then I disables ipv6 on the network card. No change.

- Then I found

https://superuser.com/questions/586144/disable-ipv6-loopback-on-windows-7-64-bit

and did [6]. No change.

I restarted the cluster with -z 127.0.0.1:10999 [6]. No change.

Any comments are very helpful.

Cheers,
Arturas

References:

[1]
F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\zkcli.bat
-zkhost 127.0.0.1:10999 -cmd downconfig -confdir d:\de_wiki_man_config2
-confname de_wiki_man

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>rem @echo off

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM You can override
pass the following parameters to this script:

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>set JVM=java

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM  Find location
of this script

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>set
SDIR=f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>if "\" == "\" set
SDIR=f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>if defined
LOG4J_PROPS (set "LOG4J_CONFIG=file:" )  else (set
"LOG4J_CONFIG=file:f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\log4j.properties"
)

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM Settings for ZK
ACL

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM set
SOLR_ZK_CREDS_AND_ACLS=-DzkACLProvider=org.apache.solr.common.cloud.VMParamsAllAndReadonlyDigestZkACLProvider
^

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
-DzkCredentialsProvider=org.apache.solr.common.cloud.VMParamsSingleSetCredentialsDigestZkCredentialsProvider
^

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
-DzkDigestUsername=admin-user -DzkDigestPassword=CHANGEME-ADMIN-PASSWORD ^

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM
-DzkDigestReadonlyUsername=readonly-user
-DzkDigestReadonlyPassword=CHANGEME-READONLY-PASSWORD

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM "java"
-Dlog4j.configuration="file:f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\log4j.properties"
^

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>REM -classpath
"f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\solr-webapp\webapp\WEB-INF\lib\*;f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\ext\*;f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\*"
org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:10999 -cmd downconfig
-confdir d:\de_wiki_man_config2 -confname de_wiki_man

F:\solr_server\solr-7.2.1\server\scripts\cloud-scripts>"java"
-Dlog4j.configuration="file:f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\log4j.properties"
-Djava.net.preferIPv4Stack=true -classpath
"f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\solr-webapp\webapp\WEB-INF\lib\*;f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\ext\*;f:\solr_server\solr-7.2.1\server\scripts\cloud-scripts\..\..\lib\*"
org.apache.solr.cloud.ZkCLI -zkhost 127.0.0.1:10999 -cmd downconfig
-confdir d:\de_wiki_man_config2 -confname de_wiki_man
Exception in thread "main" org.apache.solr.common.SolrException:
java.io.IOException: Couldn't instantiate
org.apache.zookeeper.ClientCnxnSocketNIO
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:170)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:119)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:109)
at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:197)
Caused by: java.io.IOException: Couldn't instantiate
org.apache.zookeeper.ClientCnxnSocketNIO
at
org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1842)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:447)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:380)
at
org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:43)
at
org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
at
org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:148)
... 3 

solr allow read permission to anonymous/guest user

[Dinesh Sundaram]

Hi,

Is there any option to allow read permissions to anonymous/guest user?
expecting to prompt credentials only if any update or delete operations.


Thanks,
Dinesh Sundaram

RE: Collection reload leaves dangling SolrCore instances

[Markus Jelsma]

Hello,

It doesn't appear to be fixed on 7.4. During core reload there is the memory 
spike, afterwards i loose about 110 MB in OldGen. Forcing GC doesn't free the 
memory.

On reloads, it seems one instance per core is retained. So on my local box 
where all collections are in one instance, i can easily get OOME if i reload a 
couple of replica's.

So, do you have another already existing issue for me matching this problem? Or 
shall i open a new issue.

Many thanks,
Markus



 
 
-Original message-
> From:Markus Jelsma 
> Sent: Wednesday 2nd May 2018 18:01
> To: solr-user@lucene.apache.org; solr-user 
> Subject: RE: Collection reload leaves dangling SolrCore instances
> 
> Sounds just like it, i will check it out!
> 
> Thanks both!
> Markus
> 
>  
>  
> -Original message-
> > From:Erick Erickson 
> > Sent: Wednesday 2nd May 2018 17:21
> > To: solr-user 
> > Subject: Re: Collection reload leaves dangling SolrCore instances
> > 
> > Markus:
> > 
> > You may well be hitting SOLR-11882.
> > 
> > On Wed, May 2, 2018 at 8:18 AM, Shawn Heisey  wrote:
> > > On 5/2/2018 4:40 AM, Markus Jelsma wrote:
> > >> One of our collections, that is heavy with tons of TokenFilters using 
> > >> large dictionaries, has a lot of trouble dealing with collection reload. 
> > >> I removed all custom plugins from solrconfig, dumbed the schema down and 
> > >> removed all custom filters and replaced a customized decompounder with 
> > >> Lucene's vanilla filter, and the problem still exists.
> > >>
> > >> After collection reload a second SolrCore instance appears for each real 
> > >> core in use, each next reload causes the number of instances to grow. 
> > >> The dangling instances are eventually removed except for one or two. 
> > >> When working locally with for example two shards/one replica in one JVM, 
> > >> a single reload eats about 500 MB for each reload.
> > >>
> > >> How can we force Solr to remove those instances sooner? Forcing a GC 
> > >> won't do it so it seems Solr itself actively keeps some stale instances 
> > >> alive.
> > >
> > > Custom plugins, which you did mention, would be the most likely
> > > culprit.  Those sometimes have bugs where they don't properly close
> > > resources.  Are you absolutely sure that there is no custom software
> > > loading at all?  Removing the jars entirely (not just the config that
> > > might use the jars) might be required.
> > >
> > > Have you been able to get heap dumps and figure out what object is
> > > keeping the SolrCore alive?
> > >
> > > Thanks,
> > > Shawn
> > >
> > 
> 

Re: A user defined request handler is failing to fetch the data.

[Erick Erickson]

At a glance, you may really be seeing recursive calls. What does the
solr log (rather than the client) say is happening?

Any call that comes into /filesearch will make a call to...
./filesearch perhaps?

What is your goal here? You say you're "testing distributed search".
But what do you mean by that? There are two
bits here:

1> you really mean the entire index is contained in a single core and
you want multiple cores to handle increased QPS. This isn't done by
messing with the shards parameter, since each core is a complete
index. People usually just put a load balancer in front of the slaves
(if using master/slave) or just use SolrCloud with a number of
replicas and let Solr do the work.

2> You want to use a sharded solution because you have too much data
to be handled by a single core. In that case I would _strongly_
recommend you just use SolrCloud. That would "just work" without this
kind of tweaking.

Best,
Erick

On Wed, Jun 27, 2018 at 11:01 PM, Adarsh_infor  wrote:
> Hi All,
>
> I was running SOLR 4.6.1 in master slave architecture on Windows machine,
> Now am planning to migrate that to Linux and upgrade the SOLR 6.6.3
> version(Remember only the Configs and schema will be migtrated not data). In
> the process of doing so i had to do some changes to Schema and Config in
> order to create a core without an issue, after creating the core i was
> getting a warning like the LUCENE_40 support will be removed from 7 so need
> to change it. So in order to fix that warning i have changed it
> `6.6.3`.  After which using *DIH*
> i indexed certain documents and was able to query them by using default
> `/select` request handler. But i wanted to test the distributed search so in
> order to test that . i have created one more core in the same server, which
> is having a below requestHandler defined in config.xml
>
> 
>
>*:*
>localhost:8983/solr/FI_idx
>document
>
> 
>
> Core *FI_idx* is the core which is having the indexed data in it, with
> `6.6.3` but when query using
> requestHandler `/filesearch` and getting below error could anyone help to me
> to understand whats happening.
>
>
>
> "trace":
> "org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
> from server at http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx: Error from server at
> http://localhost:8983/solr/FI_idx:
> org.apache.solr.client.solrj.SolrServerException: IOException occured when
> talking to server at: http://localhost:8983/solr/FI_idx\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)\n\tat
> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)\n\tat
> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)\n\tat
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:218)\n\tat
> org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
> org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)\n\tat
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
> java.lang.Thread.run(Thread.java:745)\n",
> "code": 500
>
>
>
> --
> Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

RE: 7.4.0 changes in DocTransformer behaviour

[Markus Jelsma]

Hello Shawn,

Ah, forgot about that thread. But yes, it makes sense and we used it as such in 
production. But the test was never updated, never failed, until 7.4.0 came 
around. So it is confusing why the tests didn't fail, and confusing why it does 
in 7.4.0. Any ideas why now?

Thanks,
Markus 
 
-Original message-
> From:Shawn Heisey 
> Sent: Wednesday 27th June 2018 17:40
> To: solr-user@lucene.apache.org
> Subject: Re: 7.4.0 changes in DocTransformer behaviour
> 
> On 6/27/2018 8:29 AM, Markus Jelsma wrote:
> > I am attempting an upgrade to 7.4.0, besides a decent amount of thread leak 
> > warnings i suddenly get, there are some strange failing tests, of which one 
> > is one of four DocTransformer plugins. That plugin is really simple, it 
> > just transformers integer or long numerics into a binary string of 0 and 1.
> >
> > This is the test:
> >
> > assertQ( req("q", "*:*", "indent", "true", "fl", 
> > "id,minhash:[binstr]"), 
> > "//*[@numFound='" + max + "']",
> > 
> > "//str[.='']",
> > 
> > "//str[.='0001']",
> > 
> > "//str[.='0010']",
> > 
> > "//str[.='0011']");
> >   }
> >
> > Changing the fl parameter into "*,minhash:[binstr]" solves the problem, the 
> > test passes.
> 
> I will admit that I do not know exactly what that assert syntax means,
> but I'm guessing that it looks for those strings of numbers in the
> response, and makes sure that numFound is what you expect it to be.
> 
> This problem sounds exactly the same as the problem you were
> encountering and asked the list about in December 2016, on version 6.3.0
> - the transformed field you've requested with fl doesn't work unless you
> also request all fields.  Here's that thread on Nabble:
> 
> http://lucene.472066.n3.nabble.com/DocTransformer-not-always-working-td4309785.html
> 
> I do not really understand what Hoss told you in response to that thread
> ... but can you use the same information this time?
> 
> Thanks,
> Shawn
> 
> 

Re: cannot download the config directory with zkcli.bat from embedded zookeeper

[Shawn Heisey]

On 6/28/2018 4:03 AM, Arturas Mazeika wrote:
> f:\solr_server\solr-7.3.1\server\scripts\cloud-scripts\zkcli.bat -zkhost
> localhost:10999 -cmd downconfig -confdir d:\de_wiki_man_config -confname
> de_wiki_man
>
> Caused by: java.io.IOException: Unable to establish loopback connection

Take a look at this SO answer (for android software, not Solr):

https://stackoverflow.com/a/42548312/2665648

You can't use that exact solution, because the _JAVA_OPTIONS environment
variable is not used by Solr or its startup script.  But I think the
underlying problem might be what you're running into:  There may be a
problem with the ipv6 setup on your system.

If you change the address in the command from localhost to 127.0.0.1,
that might fix the problem.

Setting the system property to prefer IPv4 addresses would need to
happen in the zkcli script to fix this command.

Or you can fix the network stack in Windows.  Either set up IPv6
properly, or remove IPv6 entirely.  If you do not actually have ipv6 in
your network, you might try this command on a command prompt that has
privilege elevation:

netsh int ipv6 reset ipv6reset.log

A reboot might be required after that command.

What happens if you type 'ping localhost' from a command prompt?  This
is what I get, indicating that the local ipv6 address is working:

C:\Users\sheisey>ping localhost

Pinging purgatory.REDACTED.com [::1] with 32 bytes of data:
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms
Reply from ::1: time<1ms

Thanks,
Shawn

Re: Indexing Approach

[Shawn Heisey]

On 6/27/2018 8:59 PM, solrnoobie wrote:
> One last thing though, will a queue based system work for us? Or are there
> better suitable implementations?

Exactly how you write your indexing software is up to you.  There is no
single approach that's the best.  Examine your business needs and the
things that the person who must write the program is comfortable using. 
Pick an approach that fits those.

Thanks,
Shawn

Re: Solr Update

[Shawn Heisey]

On 6/27/2018 10:00 AM, Goston IV, Stanley W wrote:
> I was wondering if it is possible to do a solr update by doing something like 
> the following:
>
> JSON:
>
> {
>   "add":{
> "doc":{
>   "id_txt":"_query_:{q=title_txt:test.jpg fl=id_txt}",
>   "title_txt": {"set":"Maybe"},
>   "location_geo": {"set":"POINT(-108.5491667 57.7119)"}
> }
>   }
> }

> I need to be able to do a nested query to get the unique id since I do not 
> know it without performing a query.

As far as I am aware, this is not possible.  Others probably know for
sure.  I have never heard of anything like this.

You will probably need to do the query in whatever software is building
the update requests before that part of the update is built.

Because your example update sets the field you're querying on to a new
value, that same query executed again isn't going to work after the
change is committed.

Thanks,
Shawn

Re: Solr 6.1 collection backup into HDFS

[Shawn Heisey]

On 1/7/2017 3:46 AM, akalfus wrote:
> Hello,I'm trying to run a collection backup with the following command -
> /solr/admin/collections?action=BACKUP=collection_name=hdfs=/tmp/backup/=nameafter
> trying to run in in local mode and failing due to not having an NFS.(BTW - I
> manually created the folder in the other nodes and it seemed to work - every
> shard was created in its own node).So I added the repository property set to
> 'hdfs', and it seems like it is ignored. What am I missing?

I did not reply at the time this message was sent because I had
absolutely no idea.  Now somebody has revived the thread, and I've
looked a little deeper.

It is not at all clear from the documentation exactly what the
'repository' parameter should be set to.  By poking at the code, I was
able to figure out what you need to do.

The repository name that you specify in the URL must be specified in
solr.xml, with a config like the following.  The config I included below
relies on system properties to work properly, but you could specify the
values directly in solr.xml:

  
    
  ${solr.hdfs.default.backup.path}
  ${solr.hdfs.home:}
  ${solr.hdfs.confdir:}
  ${solr.hdfs.permissions.umask-mode:000}
    
  

I got that config from the Solr test class named TestHdfsBackupRestoreCore.

Once you have a complete repository config in solr.xml and restart Solr,
it should work.

Thanks,
Shawn

Re: Create an index field of type dictionary

[Shawn Heisey]

On 6/26/2018 11:48 AM, Ritesh Kumar (Avanade) wrote:
> Is it possible to create an index field of type *dictionary*. I have
> seen stringarry, datetime, bool etc. but I am looking for a field type
> like list of objects.

The types that you find in a Solr schema are just made-up sequences of
characters.  You can have a type named "bugblatter" if you want.  The
"string" type found in the example schemas could be changed so it uses a
class that's made for storing numbers.  This is not a configuration
error, although it might cause big problems with trying to use Solr,
especially if it is changed on an existing index.

In the Solr reference guide, there is a list of classes available by
default for use in a fieldType definition.

https://lucene.apache.org/solr/guide/7_3/field-types-included-with-solr.html

Erick has raised a good point.  What do you want to a dictionary field
type to *do*?

It looks like Sitecore is a large piece of software that happens to
include Solr.  Chances are good that they have customized their Solr
install and they will be in the best position to help you with it.

Thanks,
Shawn

cannot download the config directory with zkcli.bat from embedded zookeeper

[Arturas Mazeika]

Hi Solr Folks,

I was benchmarking solrCloud with the German Wikipedia docs under Windows
10 Pro, noticed that Windows defender was grabbing resources substantially,
changed the policy (so windows defender would not kick in). Since then I am
not able to download the config files from the solrCloud with

f:\solr_server\solr-7.3.1\server\scripts\cloud-scripts\zkcli.bat -zkhost
localhost:10999 -cmd downconfig -confdir d:\de_wiki_man_config -confname
de_wiki_man


Caused by: java.io.IOException: Unable to establish loopback connection

(see [1] for more details)

What's going on? On a Linux VM the same commands fly with green colors.

Details:

An example cloud environment was created with

F:\solr_server\solr-7.3.1>bin\solr.cmd start -e cloud

(see [2] for more details)

Netstat on windows reports no problems [3].

I am able to connect to 10999 port with telnet and get the stats [4].

solrs can communicate with each other without problems [5].

zoo.cfg is the standart one [6].

I also tried the same with 7.2.1 and 7.4.0 (without any success and the
same error). Interestingly enough, there's an error in script files in
7.4.0 (no biggy) [7].

Cheers,
Arturas

References:

[1] F:\solr_server\solr-7.3.1>server\scripts\cloud-scripts\zkcli.bat
-zkhost localhost:10999 -cmd downconfig -confdir d:\de_wiki_man_config2
-confname de_wiki_man
Exception in thread "main" org.apache.solr.common.SolrException:
java.io.IOException: Couldn't instantiate
org.apache.zookeeper.ClientCnxnSocketNIO
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:171)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:120)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:110)
at org.apache.solr.cloud.ZkCLI.main(ZkCLI.java:197)
Caused by: java.io.IOException: Couldn't instantiate
org.apache.zookeeper.ClientCnxnSocketNIO
at
org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1845)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:452)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:383)
at
org.apache.solr.common.cloud.SolrZooKeeper.(SolrZooKeeper.java:43)
at
org.apache.solr.common.cloud.ZkClientConnectionStrategy.createSolrZooKeeper(ZkClientConnectionStrategy.java:105)
at
org.apache.solr.common.cloud.DefaultConnectionStrategy.connect(DefaultConnectionStrategy.java:37)
at
org.apache.solr.common.cloud.SolrZkClient.(SolrZkClient.java:149)
... 3 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Source)
at java.lang.reflect.Constructor.newInstance(Unknown Source)
at
org.apache.zookeeper.ZooKeeper.getClientCnxnSocket(ZooKeeper.java:1843)
... 9 more
Caused by: java.io.IOException: Unable to establish loopback connection
at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
at sun.nio.ch.PipeImpl$Initializer.run(Unknown Source)
at java.security.AccessController.doPrivileged(Native Method)
at sun.nio.ch.PipeImpl.(Unknown Source)
at sun.nio.ch.SelectorProviderImpl.openPipe(Unknown Source)
at java.nio.channels.Pipe.open(Unknown Source)
at sun.nio.ch.WindowsSelectorImpl.(Unknown Source)
at sun.nio.ch.WindowsSelectorProvider.openSelector(Unknown Source)
at java.nio.channels.Selector.open(Unknown Source)
at
org.apache.zookeeper.ClientCnxnSocketNIO.(ClientCnxnSocketNIO.java:43)
... 14 more
Caused by: java.net.ConnectException: Connection timed out: connect
at sun.nio.ch.Net.connect0(Native Method)
at sun.nio.ch.Net.connect(Unknown Source)
at sun.nio.ch.Net.connect(Unknown Source)
at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
at java.nio.channels.SocketChannel.open(Unknown Source)
at sun.nio.ch.PipeImpl$Initializer$LoopbackConnector.run(Unknown
Source)


[2] F:\solr_server\solr-7.3.1>bin\solr.cmd start -e cloud

Welcome to the SolrCloud example!

This interactive session will help you launch a SolrCloud cluster on your
local workstation.
To begin, how many Solr nodes would you like to run in your local cluster?
(specify 1-4 nodes) [2]:
4
Ok, let's start up 4 Solr nodes for your example SolrCloud cluster.
Please enter the port for node1 [8983]:

Please enter the port for node2 [7574]:
9998
Please enter the port for node3 [8984]:
9997
Please enter the port for node4 [7575]:
9996
Creating Solr home directory
F:\solr_server\solr-7.3.1\example\cloud\node1\solr
Cloning F:\solr_server\solr-7.3.1\example\cloud\node1 into
   F:\solr_server\solr-7.3.1\example\cloud\node2
Cloning F:\solr_server\solr-7.3.1\example\cloud\node1 into
   F:\solr_server\solr-7.3.1\example\cloud\node3
Cloning 

A user defined request handler is failing to fetch the data.

[Adarsh_infor]

Hi All,

I was running SOLR 4.6.1 in master slave architecture on Windows machine,
Now am planning to migrate that to Linux and upgrade the SOLR 6.6.3
version(Remember only the Configs and schema will be migtrated not data). In
the process of doing so i had to do some changes to Schema and Config in
order to create a core without an issue, after creating the core i was
getting a warning like the LUCENE_40 support will be removed from 7 so need
to change it. So in order to fix that warning i have changed it
`6.6.3`.  After which using *DIH* 
i indexed certain documents and was able to query them by using default
`/select` request handler. But i wanted to test the distributed search so in
order to test that . i have created one more core in the same server, which
is having a below requestHandler defined in config.xml 


   
   *:*
   localhost:8983/solr/FI_idx
   document
   


Core *FI_idx* is the core which is having the indexed data in it, with
`6.6.3` but when query using
requestHandler `/filesearch` and getting below error could anyone help to me
to understand whats happening.  



"trace":
"org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx: Error from server at
http://localhost:8983/solr/FI_idx:
org.apache.solr.client.solrj.SolrServerException: IOException occured when
talking to server at: http://localhost:8983/solr/FI_idx\n\tat
org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:560)\n\tat
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:235)\n\tat
org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:227)\n\tat
org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1220)\n\tat
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:218)\n\tat
org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:183)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:266)\n\tat
org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor$1.run(ExecutorUtil.java:148)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\n\tat
java.lang.Thread.run(Thread.java:745)\n",
"code": 500



--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html