Fieldname alias for Highlighter results

[Michael Aleythe, Sternwald]

Hi everybody,

I'm looking for a way to replace solr index field names in the highlighting 
response.

For the query part there is the param fl=substitute:REAL_FIELD_NAME which 
substitutes the field name REAL_FIELD_NAME by "substitute". Sadly the 
substitution is not applied to the highlighter response. I tried using the same 
syntax on the hl.fl param but with no success. Has anybody an idea on how to 
achieve this?

Best regards
Michael Aleythe

SolrResourceLoader is not closed when SolrCore is closed

[Michael Aleythe, Sternwald]

Hi everybody,

I'm currently using solr 8.3.1. While working on tests for a solr plugin I'm 
writing , I realized that the SolrResourceLoader used by a SolrCore is not 
closed, when calling SolrCore.close().
This leads to open file handles. Can anybody explain this behavior?

Best Regards,

Michael Aleythe
Team --(sr)^(ch)--
Java Entwickler | STERNWALD SYSTEMS GMBH

Fon +49 351 31 40 6010
Fax +49 351 31 40 6001

E-Mail michael.aley...@sternwald.com
Skype michael.aley...@sternwald.com
Web www.sternwald.com

STERNWALD SYSTEMS GMBH
Pohlandstraße 19, 01309 Dresden, Germany
Geschäftsführer Ard Meier
Registergericht Handelsregister Dresden, HRB 33480
UmSt-ID DE157125091

SUPPORT / HOTLINE
Fon +49 173 38 54 752
E-Mail hotl...@sternwald.com
Web support.sternwald.net

STERNWALD Offices
Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg

local paramas only with defType=lucene?

[Michael Aleythe, Sternwald]

Hi everybody,

is it correct that local parameters ( q={!edismax qf=MEDIA_ID v=283813390} ) in 
solr only work with the lucene query parser defined for the main query? I tried 
with dismax/edismax but it did not work. The documentation is not clear on this 
point.

Best regards
Michael Aleythe

AW: Keyword field with tabs in Solr 7.4

[Michael Aleythe, Sternwald]

Hey Erik,

thanks a lot for your suggestion. It lead me on the rigth path. What actually 
did the trick was, sending the tab as unicode: 
IPTC_2_080_KY:"\u0009bus\u0009bahn" matched perfectly.

Best,
Michael

-Ursprüngliche Nachricht-
Von: Erick Erickson  
Gesendet: Dienstag, 11. Dezember 2018 18:45
An: solr-user 
Betreff: Re: Keyword field with tabs in Solr 7.4

You are probably in "url-encoding hell". Add =query to your search and 
check the parsed query returned to see what Solr actually sees. Try 
url-encoding the backslash *%5C" maybe?

Best,
Erick
On Tue, Dec 11, 2018 at 1:40 AM Michael Aleythe, Sternwald 
 wrote:
>
> Hey everybody,
>
> i have a Solr field keyword field defined as:
>
> 
>  
>
>  
> 
>
>  stored="true" termVectors="false" multiValued="false" />
>
> Some documents have tabs (\t) indexed in this field, e.g. 
> IPTC_2_080_KY:"\tbus\tbahn"
>
> How can i query this content? I tried  "\tbus\tbahn", 
> \\tbus\\tbahn and " bus bahn" but nothing matches. Does 
> anybody know what to do?
>
> Regards
> Michael

Keyword field with tabs in Solr 7.4

[Michael Aleythe, Sternwald]

Hey everybody,

i have a Solr field keyword field defined as:


 
   
 




Some documents have tabs (\t) indexed in this field, e.g. 
IPTC_2_080_KY:"\tbus\tbahn"

How can i query this content? I tried  "\tbus\tbahn", 
\\tbus\\tbahn and " bus bahn" but nothing matches. Does 
anybody know what to do?

Regards
Michael

AW: Solr 7.4 and log4j2 JSONLayout

[Michael Aleythe, Sternwald]

Hey Shawn,

i removed the additional jars and tried the foreground option with the 
following result:

2018-09-10 11:41:03.174:INFO::main: Logging initialized @616ms to 
org.eclipse.jetty.util.log.StdErrLog
2018-09-10 11:41:03.313:INFO:oejs.Server:main: jetty-9.4.10.v20180503; built: 
2018-05-03T15:56:21.710Z; git: daa59876e6f384329b122929e70a80934569428c; jvm 
1.8.0_45-b14
2018-09-10 11:41:03.338:INFO:oejdp.ScanningAppProvider:main: Deployment monitor 
[file:///usr/local/dir/solr-7.4.0/server/contexts/] at interval 0
2018-09-10 11:41:03.578:INFO:oejw.StandardDescriptorProcessor:main: NO JSP 
Support for /solr, did not find org.apache.jasper.servlet.JspServlet
2018-09-10 11:41:03.586:INFO:oejs.session:main: DefaultSessionIdManager 
workerName=node0
2018-09-10 11:41:03.586:INFO:oejs.session:main: No SessionScavenger set, using 
defaults
2018-09-10 11:41:03.588:INFO:oejs.session:main: node0 Scavenging every 60ms
2018-09-10 11:41:05.091:INFO:oejsh.ContextHandler:main: Started 
o.e.j.w.WebAppContext@7c7a06ec{/solr,file:///usr/local/dir/solr-7.4.0/server/solr-webapp/webapp/,AVAILABLE}{file:///usr/local/dir/solr-7.4.0/server//solr-webapp/webapp}
2018-09-10 11:41:05.133:INFO:oejs.AbstractConnector:main: Started 
ServerConnector@10650953{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
2018-09-10 11:41:05.133:INFO:oejs.Server:main: Started @2575ms

There is absolutely no clue about whats going on.

Regarding the rollover size. I was just playing around to see if the rollover 
works as I expected.

Thanks,
Michael

-Ursprüngliche Nachricht-
Von: Shawn Heisey  
Gesendet: Freitag, 7. September 2018 17:58
An: solr-user@lucene.apache.org
Betreff: Re: Solr 7.4 and log4j2 JSONLayout

On 9/6/2018 7:46 AM, Michael Aleythe, Sternwald wrote:
> I'm trying to edit the log4j2 logging configuration for solr. The goal is to 
> get a log file in json format. I configured the the JSONLayout for this 
> purpose inside the rollingFile appender in the log4j2.xml. After this solr 
> stops logging entirely. Solr.log file is empty. Only the 
> solr-8983-console.log file contains 10 lines. The line "2018-09-06 
> 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars were 
> missing, but that did not fix the problem.

As Varun said, jackson is already included in Solr.  You won't need to add any 
jars for that.

> Does anyone know where to find error-messages or exceptions that point me 
> towards whats going wrong here?

Start Solr in the foreground, with the -f option.  This will cause Solr to log 
to the console.  When Solr is started in the background, it suppresses console 
logging.

I see that you have changed the logfile rollover size to 1MB.  If your Solr 
server sees much traffic, this is going to result in an extremely fast 
rollover, which may mean that you lose access to logged events VERY quickly.  
This will especially be the case with JSON logging -- each event will take up a 
lot more space.

Thanks,
Shawn

AW: Solr 7.4 and log4j2 JSONLayout

[Michael Aleythe, Sternwald]

Hey,

I tried solr/server/lib/ext and solr/server/lib. I also tried without them but 
it doesn't change anything.

Best regards
Michael

-Ursprüngliche Nachricht-
Von: Varun Thacker  
Gesendet: Donnerstag, 6. September 2018 16:23
An: solr-user@lucene.apache.org
Betreff: Re: Solr 7.4 and log4j2 JSONLayout

Hi,

Where did you add the jackson core and databind libs under? Maybe it's 
conflicting with the JARs that Solr already ships?

Solr already comes with jackson dependencies

server/solr-webapp/webapp/WEB-INF/lib/ | grep jackson

jackson-annotations-2.9.5.jar

jackson-core-2.9.5.jar

jackson-core-asl-1.9.13.jar

jackson-databind-2.9.5.jar

jackson-dataformat-smile-2.9.5.jar

jackson-mapper-asl-1.9.13.jar

On Thu, Sep 6, 2018 at 6:46 AM Michael Aleythe, Sternwald < 
michael.aley...@sternwald.com> wrote:

> Hey,
>
> I'm trying to edit the log4j2 logging configuration for solr. The goal 
> is to get a log file in json format. I configured the the JSONLayout 
> for this purpose inside the rollingFile appender in the log4j2.xml. 
> After this solr stops logging entirely. Solr.log file is empty. Only 
> the solr-8983-console.log file contains 10 lines. The line "2018-09-06
> 13:22:25.378:INFO:oejs.Server:main: Started @2814ms" is the last one.
> My first guess was that the jackson-core and jackson-databind jars 
> were missing, but that did not fix the problem.
>
> Does anyone know where to find error-messages or exceptions that point 
> me towards whats going wrong here?
>
> This is my current log4j2 config file:
>
>  
>   
>
> 
>   
> 
>   %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} 
> %X{shard} %X{replica} %X{core}] %c{1.} %m%n
> 
>   
> 
>
>  name="RollingFile"
> fileName="${sys:solr.log.dir}/solr.log"
>
> filePattern="${sys:solr.log.dir}/solr.log.%d{-MM-dd-hh}_%i.zip" >
>   
>   
> 
> 
> 
>   
>   
> 
>
>  name="SlowFile"
> fileName="${sys:solr.log.dir}/solr_slow_requests.log"
> filePattern="${sys:solr.log.dir}/solr_slow_requests.log.%i" >
>   
> 
>   %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} 
> %X{shard} %X{replica} %X{core}] %c{1.} %m%n
> 
>   
>   
> 
> 
>   
>   
> 
>
>   
>   
> 
> 
> 
>  additivity="false">
>   
> 
>   
>   
> 
>   
> 
>
> Best regards
> Michael Aleythe
>
> Michael Aleythe
> Team --(sr)^(ch)--
> Java Entwickler | STERNWALD SYSTEMS GMBH
>
> Fon +49 351 31 40 6010
> Fax +49 351 31 40 6001
>
> E-Mail michael.aley...@sternwald.com
> Skype michael.aley...@sternwald.com
> Web www.sternwald.com
>
> STERNWALD SYSTEMS GMBH
> Pohlandstraße 19, 01309 Dresden, Germany Geschäftsführer Ard Meier 
> Registergericht Handelsregister Dresden, HRB 33480 UmSt-ID DE157125091
>
> SUPPORT / HOTLINE
> Fon +49 173 38 54 752
> E-Mail hotl...@sternwald.com
> Web support.sternwald.net
>
> STERNWALD Offices
> Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg
>
>

Solr 7.4 and log4j2 JSONLayout

[Michael Aleythe, Sternwald]

Hey,

I'm trying to edit the log4j2 logging configuration for solr. The goal is to 
get a log file in json format. I configured the the JSONLayout for this purpose 
inside the rollingFile appender in the log4j2.xml. After this solr stops 
logging entirely. Solr.log file is empty. Only the solr-8983-console.log file 
contains 10 lines. The line "2018-09-06 13:22:25.378:INFO:oejs.Server:main: 
Started @2814ms" is the last one.
My first guess was that the jackson-core and jackson-databind jars were 
missing, but that did not fix the problem.

Does anyone know where to find error-messages or exceptions that point me 
towards whats going wrong here?

This is my current log4j2 config file:



  


  

  %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} 
%X{replica} %X{core}] %c{1.} %m%n

  



  
  



  
  



  

  %d{-MM-dd HH:mm:ss.SSS} %-5p (%t) [%X{collection} %X{shard} 
%X{replica} %X{core}] %c{1.} %m%n

  
  


  
  


  
  




  

  
  

  


Best regards
Michael Aleythe

Michael Aleythe
Team --(sr)^(ch)--
Java Entwickler | STERNWALD SYSTEMS GMBH

Fon +49 351 31 40 6010
Fax +49 351 31 40 6001

E-Mail michael.aley...@sternwald.com
Skype michael.aley...@sternwald.com
Web www.sternwald.com

STERNWALD SYSTEMS GMBH
Pohlandstraße 19, 01309 Dresden, Germany
Geschäftsführer Ard Meier
Registergericht Handelsregister Dresden, HRB 33480
UmSt-ID DE157125091

SUPPORT / HOTLINE
Fon +49 173 38 54 752
E-Mail hotl...@sternwald.com
Web support.sternwald.net

STERNWALD Offices
Berlin | Dresden | Düsseldorf | Hamburg | Sofia | Würzburg

AW: AW: Resync after restart

[Michael Aleythe, Sternwald]

You need to change this on the master node. The part of the config you pasted 
here, looks like it is from the slave node.

-Ursprüngliche Nachricht-
Von: Arkadi Colson [mailto:ark...@smartbit.be] 
Gesendet: Donnerstag, 24. November 2016 11:56
An: solr-user@lucene.apache.org
Betreff: Re: AW: Resync after restart

Hi Michael

Thanks for the quick response! The line does not exist in my config. So can I 
assume that the default configuration is to not replicate at startup?

   
 
   18.75
   05:00:00
   15
   30
 
   

Any other idea's?


On 24-11-16 11:49, Michael Aleythe, Sternwald wrote:
> Hi Arkadi,
>
> you need to remove the line "startup" from 
> your ReplicationHandler-config in solrconfig.xml -> 
> https://wiki.apache.org/solr/SolrReplication.
>
> Greetings
> Michael
>
> -Ursprüngliche Nachricht-
> Von: Arkadi Colson [mailto:ark...@smartbit.be]
> Gesendet: Donnerstag, 24. November 2016 09:26
> An: solr-user <solr-user@lucene.apache.org>
> Betreff: Resync after restart
>
> Hi
>
> Almost every time when restarting a solr instance the index is replicated 
> completely. Is there a way to avoid this somehow? The index currently has a 
> size of about 17GB.
> Some advice here would be great.
>
> 99% of the config is defaul:
>
>  ${solr.ulog.dir:}  name="numVersionBuckets">${solr.ulog.numVersionBuckets:65536}
>  
>  ${solr.autoCommit.maxTime:15000}
>  false
>
>
> If you need more info, just let me know...
>
> Thx!
> Arkadi
>

AW: Resync after restart

[Michael Aleythe, Sternwald]

Hi Arkadi,

you need to remove the line "startup" from 
your ReplicationHandler-config in solrconfig.xml -> 
https://wiki.apache.org/solr/SolrReplication.

Greetings
Michael

-Ursprüngliche Nachricht-
Von: Arkadi Colson [mailto:ark...@smartbit.be] 
Gesendet: Donnerstag, 24. November 2016 09:26
An: solr-user 
Betreff: Resync after restart

Hi

Almost every time when restarting a solr instance the index is replicated 
completely. Is there a way to avoid this somehow? The index currently has a 
size of about 17GB.
Some advice here would be great.

99% of the config is defaul:

 ${solr.ulog.dir:} ${solr.ulog.numVersionBuckets:65536}
   
${solr.autoCommit.maxTime:15000}  
false  
  

If you need more info, just let me know...

Thx!
Arkadi

json facet api and facet.threads

[Michael Aleythe, Sternwald]

Hi Everybody,

can anyone point me in the right direction for using "facet.threads" with the 
json faceting-api? Does it only work if terms facets are exclusively used in 
the query?

Best regards

Michael Aleythe
Java Entwickler | STERNWALD SYSTEMS GMBH

json request api and facets

[Michael Aleythe, Sternwald]

Hi everybody,

i'm currently working on using the json request api for solr and hit a problem 
using facets. I'm using solr 5.5.2 and solrJ 5.5.2

When querying solr by url-parameters like so:
http://.../select?wt=json=MEDIA_TS_TS.facet.range.end=2028-02-01T0:00:00.000Z_TS.facet.range.gap=%2B1YEAR_TS.facet.range.start=1993-01-01T0:00:00.000Z=true

the returned json contains an element called "facet_counts" which is the top 
element for all faceting information.


:   "facet_counts":
:   {
:   :   "facet_queries":
:   :   {
:   :   },
:   :   "facet_fields":
:   :   {
:   :   },
:   :   "facet_dates":
:   :   {
:   :   },
:   :   "facet_ranges":
:   :   {
:   :   :   "MEDIA_TS":
:   :   :   {
:   :   :   :   "counts":
:   :   :   :   [
:   :   :   :   :   "1993-01-01T00:00:00Z",
:   :   :   :   :   0,
:   :   :   :   :   "1994-01-01T00:00:00Z",
:   :   :   :   :   1634,
:   :   :   :   :   "1995-01-01T00:00:00Z",
:   :   :   :   :   6656,
:   :   :   :   :   "1996-01-01T00:00:00Z",
:   :   :   :   :   30016,
:   :   :   :   :   "1997-01-01T00:00:00Z",
:   :   :   :   :   76819,
:   :   :   :   :   "1998-01-01T00:00:00Z",
:   :   :   :   :   152099,

The same query using the json request api like so:

{"facet":{"MEDIA_TS":{"field":"MEDIA_TS","gap":"+1YEAR","start":"1993-01-01T00:00:00Z","end":"2028-01-01T00:00:00Z","type":"range"}}}

Returns an element "facets" which is the top element for all faceting 
information. The whole structure of the response is different:

:   "facets":
:   {
:   :   "count":5815481,
:   :   "MEDIA_TS":
:   :   {
:   :   :   "buckets":
:   :   :   [
:   :   :   :   {
:   :   :   :   :   "val":"1993-01-01T00:00:00Z",
:   :   :   :   :   "count":0
:   :   :   :   },
:   :   :   :   {
:   :   :   :   :   "val":"1994-01-01T00:00:00Z",
:   :   :   :   :   "count":1634
:   :   :   :   },
:   :   :   :   {
:   :   :   :   :   "val":"1995-01-01T00:00:00Z",
:   :   :   :   :   "count":6656
:   :   :   :   },

This inconsistency breaks the respone parser of solrJ. Am i doing something 
wrong?


Best Regards
Michael


Michael Aleythe
Java Entwickler | STERNWALD SYSTEMS GMBH

solr json facets return format

[Michael Aleythe, Sternwald]

Hi everybody,

i'm currently working on using the json request api for solr and hit a problem 
using facets. I'm using solr 5.5.2 and solrJ 5.5.2

When querying solr by url-parameters like so:
http://.../select?wt=json=MEDIA_TS_TS.facet.range.end=2028-02-01T0:00:00.000Z_TS.facet.range.gap=%2B1YEAR_TS.facet.range.start=1993-01-01T0:00:00.000Z=true

the returned json contains an element called "facet_counts" which is the top 
element for all faceting information.


:   "facet_counts":
:   {
:   :   "facet_queries":
:   :   {
:   :   },
:   :   "facet_fields":
:   :   {
:   :   },
:   :   "facet_dates":
:   :   {
:   :   },
:   :   "facet_ranges":
:   :   {
:   :   :   "MEDIA_TS":
:   :   :   {
:   :   :   :   "counts":
:   :   :   :   [
:   :   :   :   :   "1993-01-01T00:00:00Z",
:   :   :   :   :   0,
:   :   :   :   :   "1994-01-01T00:00:00Z",
:   :   :   :   :   1634,
:   :   :   :   :   "1995-01-01T00:00:00Z",
:   :   :   :   :   6656,
:   :   :   :   :   "1996-01-01T00:00:00Z",
:   :   :   :   :   30016,
:   :   :   :   :   "1997-01-01T00:00:00Z",
:   :   :   :   :   76819,
:   :   :   :   :   "1998-01-01T00:00:00Z",
:   :   :   :   :   152099,

The same query using the json request api like so:

{"facet":{"MEDIA_TS":{"field":"MEDIA_TS","gap":"+1YEAR","start":"1993-01-01T00:00:00Z","end":"2028-01-01T00:00:00Z","type":"range"}}}

Returns an element "facets" which is the top element for all faceting 
information:

:   "facets":
:   {
:   :   "count":5815481,
:   :   "MEDIA_TS":
:   :   {
:   :   :   "buckets":
:   :   :   [
:   :   :   :   {
:   :   :   :   :   "val":"1993-01-01T00:00:00Z",
:   :   :   :   :   "count":0
:   :   :   :   },
:   :   :   :   {
:   :   :   :   :   "val":"1994-01-01T00:00:00Z",
:   :   :   :   :   "count":1634
:   :   :   :   },
:   :   :   :   {
:   :   :   :   :   "val":"1995-01-01T00:00:00Z",
:   :   :   :   :   "count":6656
:   :   :   :   },

This inconsistency breaks the respone parser of solrJ. Am i doing something 
wrong?


Best Regards


Michael Aleythe
Java Entwickler | STERNWALD SYSTEMS GMBH