Re: Investigating Seeming Deadlock

[Mike Drob]

Were you having any OOM errors beforehand? If so, that could have caused
some GC of objects that other threads still expect to be reachable, leading
to these null monitors.

On Fri, Mar 5, 2021 at 12:55 PM Stephen Lewis Bianamara <
stephen.bianam...@gmail.com> wrote:

> Hi SOLR Community,
>
> I'm investigating a node on solr 8.3.1 running in cloud mode which appears
> to have deadlocked, and I'm trying to figure out if this is a known issue
> or not, and looking for some guidance in understanding both (a) whether
> this is a resolved issue in future releases or needs a bug, and (b) how to
> lower the risk of recurrence until it is fixed.
>
> Here is what I've observed:
>
>- strace shows the main process waiting. A spot check on child processes
>shows the same, though I did not deep dive all of the threads yet (there
>are over 100).
>- the server was not doing anything or busy, except for jvm sitting at
>constant memory usage. No resource of memory, swap, cpu, etc... was
> limited
>or showing active usage.
>- jcmd Thread.Print shows some interesting info which suggests a
>deadlock or another type of locking issue
>   - For example, I found this log suggests something unusual because it
>   looks like it's trying to lock a null object
>  - "Finalizer" #3 daemon prio=8 os_prio=0 cpu=11.11ms
>  elapsed=11.11s tid=0x0100 nid=0x in
> Object.wait()
>   [0x1000]
> java.lang.Thread.State: WAITING (on object monitor)
>  at java.lang.Object.wait(java.base@11.0.7/Native Method)
>  - waiting on 
>  at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7
>  /ReferenceQueue.java:155)
>  - waiting to re-lock in wait() <0x00020020> (a
>  java.lang.ref.ReferenceQueue$Lock)
>  at java.lang.ref.ReferenceQueue.remove(java.base@11.0.7
>  /ReferenceQueue.java:176)
>  at
>  java.lang.ref.Finalizer$FinalizerThread.run(java.base@11.0.7
>  /Finalizer.java:170)
>  - I also see a lot of this. Some addressess occur multiple times,
>   but one in particular occurs 31 times. Maybe related?
>  - "h2sc-1-thread-11" #110 prio=5 os_prio=0 cpu=54.29ms
>  elapsed=11.11s tid=0x10010100 nid=0x waiting
> on condition
>   [0x10011000]
> java.lang.Thread.State: WAITING (parking)
>  at jdk.internal.misc.Unsafe.park(java.base@11.0.7/Native
>  Method)
>  - parking to wait for  <0x00030033>
>
> Can anyone help answer whether this is known or what I could look at next?
>
> Thanks!
> Stephen
>

Re: Partial update bug on solr 8.8.0

[Mike Drob]

This looks like a bug that is already fixed but not yet released in 8.9

https://issues.apache.org/jira/plugins/servlet/mobile#issue/SOLR-13034

On Tue, Mar 2, 2021 at 6:27 AM Mohsen Saboorian  wrote:

> Any idea about this post?
> https://stackoverflow.com/q/66335803/141438
>
> Regards.
>

Re: Asymmetric Key Size not sufficient

[Mike Drob]

Future vulnerability reports should be sent to secur...@apache.org so that
they can be resolved privately.

Thank you

On Fri, Feb 12, 2021 at 10:17 AM Ishan Chattopadhyaya <
ichattopadhy...@gmail.com> wrote:

> Recent versions of Solr use 2048.
>
> https://github.com/apache/lucene-solr/blob/branch_8_6/solr/core/src/java/org/apache/solr/util/CryptoKeys.java#L332
>
> Thanks for your report.
>
> On Fri, Feb 12, 2021 at 3:44 PM Mahir Kabir  wrote:
>
> > Hello,
> >
> > I am a Ph.D. student at Virginia Tech, USA. While working on a security
> > project-related work, we came across the following vulnerability in the
> > source code -
> >
> > In file
> >
> >
> https://github.com/apache/lucene-solr/blob/branch_6_6/solr/core/src/java/org/apache/solr/util/CryptoKeys.java
> > <
> >
> https://github.com/apache/ranger/blob/71e1dd40366c8eb8e9c498b0b5158d85d603af02/kms/src/main/java/org/apache/hadoop/crypto/key/RangerKeyStore.java
> > >
> > (at
> > Line 300) Key Size was set as 1024.
> >
> > *Security Impact*:
> >
> > < 2048 key size for RSA algorithm makes the system vulnerable to
> > brute-force attack
> >
> > *Useful resource*:
> > https://rules.sonarsource.com/java/type/Vulnerability/RSPEC-4426
> > https://rules.sonarsource.com/java/type/Vulnerability/RSPEC-4426
> >
> > *Solution we suggest*:
> >
> > For RSA algorithm, the key size should be >= 2048
> >
> > *Please share with us your opinions/comments if there is any*:
> >
> > Is the bug report helpful?
> >
> > Please let us know what you think about the issue. Any feedback will be
> > appreciated.
> >
> > Thank you,
> > Md Mahir Asef Kabir
> > Ph.D. Student
> > Department of CS
> > Virginia Tech
> >
>

Re: Ghost Documents or Shards out of Sync

[Mike Drob]

To expand on what Jason suggested, if the issue is the non-deterministic
ordering due to staggered commits per replica, you may have more
consistency with TLOG replicas rather than the NRT replicas. In this case,
the underlying segment files should be identical and lead to more
predictable results.

On Mon, Feb 1, 2021 at 2:50 PM Jason Gerlowski 
wrote:

> Hi Ronen,
>
> The first thing I'd figure out in your situation is whether the
> results are actually different each time, or whether the ordering is
> what differs (which might push a particular result off the page you're
> looking at, giving the appearance that it didn't match).
>
> In the case of the former, this can happen briefly if queries come in
> when some but not all replicas have seen a commit.  But usually this
> is a transient concern - either waiting for the next autocommit or
> triggering an explicit commit resolves the discrepancy in this case.
> Since you only see identical results after a restart, this _doesn't_
> sound like what you're seeing.
>
> In the case of the latter (same results, differently ordered) this is
> expected sometimes.  Solr sorts on relevance by default with the
> internal Lucene document ID being a tiebreaker.  Both the relevance
> statistics and Lucene's document IDs can differ across SolrCloud
> replicas (due to non-deterministic conditions such as the segment
> merging and deleted-doc removal that Lucene does under the hood), and
> this can produce differently-ordered result sets for users that issue
> the same query repeatedly.
>
> Good luck narrowing things down!
>
> Jason
>
> On Mon, Jan 25, 2021 at 3:32 AM Ronen Nussbaum  wrote:
> >
> > Hi All,
> >
> > I'm using Solr Cloud (version 8.3.0) with shards and replicas
> (replication
> > factor of 2).
> > Recently, I've encountered several times that running the same query
> > repeatedly yields different results. Restarting the nodes fixes the
> problem
> > (until next time).
> > I assume that some shards are not synchronized and I have several
> questions:
> > 1. What can cause this - many atomic updates? issues with commits?
> > 2. Can I trigger the "fixing" mechanism that Solr runs at restart by an
> API
> > call or some other method?
> >
> > Thanks in advance,
> > Ronen.
>

Re: Apache Solr Reference Guide isn't accessible

[Mike Drob]

Hi Dorion,

We are currently working with our infra team to get these restored. In the
meantime, the 8.4 guide is still available at
https://lucene.apache.org/solr/guide/8_4/ and are hopeful that the 8.8
guide will be back up soon. Thank you for your patience.

Mike

On Mon, Feb 1, 2021 at 1:58 PM Dorion Caroline 
wrote:

> Hi,
>
> I can't access to Apache Solr Reference Guide since few days.
> Example:
> URL
>
>   *   https://lucene.apache.org/solr/guide/8_8/
>   *   https://lucene.apache.org/solr/guide/8_7/
> Result:
> Not Found
> The requested URL was not found on this server.
>
> Do you know what going on?
>
> Thanks
> Caroline Dorion
>

Re: Solr 8.7.0 memory leak?

[Mike Drob]

Are you running these in docker containers?

Also, I’m assuming this is a typo but just in case the setting is Xmx :)

Can you share the OOM stack trace? It’s not always running out of memory,
sometimes Java throws OOM for file handles or threads.

Mike

On Wed, Jan 27, 2021 at 10:00 PM Luke  wrote:

> Shawn,
>
> it's killed by OOME exception. The problem is that I just created empty
> collections and the Solr JVM keeps growing and never goes down. there is no
> data at all. at the beginning, I set Xxm=6G, then 10G, now 15G, Solr 8.7
> always use all of them and it will be killed by oom.sh once jvm usage
> reachs 100%.
>
> I have another solr 8.6.2 cloud(3 nodes) in separated environment , which
> have over 100 collections, the Xxm = 6G , jvm is always 4-5G.
>
>
>
> On Thu, Jan 28, 2021 at 2:56 AM Shawn Heisey  wrote:
>
> > On 1/27/2021 5:08 PM, Luke Oak wrote:
> > > I just created a few collections and no data, memory keeps growing but
> > never go down, until I got OOM and solr is killed
> > >
> > > Any reason?
> >
> > Was Solr killed by the operating system's oom killer or did the death
> > start with a Java OutOfMemoryError exception?
> >
> > If it was the OS, then the entire system doesn't have enough memory for
> > the demands that are made on it.  The problem might be Solr, or it might
> > be something else.  You will need to either reduce the amount of memory
> > used or increase the memory in the system.
> >
> > If it was a Java OOME exception that led to Solr being killed, then some
> > resource (could be heap memory, but isn't always) will be too small and
> > will need to be increased.  To figure out what resource, you need to see
> > the exception text.  Such exceptions are not always recorded -- it may
> > occur in a section of code that has no logging.
> >
> > Thanks,
> > Shawn
> >
>

Re: NullPointerException in Graph Traversal nodes streaming expression

[Mike Drob]

Can you provide a sample expression that would be able to reproduce this?
Are you able to try a newer version by chance - I know we've fixed a few
NPEs recently, maybe https://issues.apache.org/jira/browse/SOLR-14700

On Thu, Jan 21, 2021 at 4:13 PM ufuk yılmaz 
wrote:

> Solr version 8.4. I’m getting an unexplanetory NullPointerException when
> executing a simple 2 level nodes stream, do you have any idea what may
> cause this?
>
> I tried setting /stream?partialResults=true&shards.tolerant=true and
> shards.tolerant=true in nodes expressions, with no luck. I also tried
> reading source of GatherNodesStream in branch 8_4, but couldn’t understand
> it. Here is a beautiful stack trace:
>
> solr| 2021-01-21 22:00:12.726 ERROR (qtp832292933-25149)
> [c:WorkerCollection s:shard1 r:core_node10
> x:WorkerCollection_shard1_replica_n9] o.a.s.c.s.i.s.ExceptionStream
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: java.lang.NullPointerException
> solr|   at
> org.apache.solr.client.solrj.io.graph.GatherNodesStream.read(GatherNodesStream.java:607)
> solr|   at
> org.apache.solr.client.solrj.io.stream.ExceptionStream.read(ExceptionStream.java:71)
> solr|   at
> org.apache.solr.handler.StreamHandler$TimerStream.read(StreamHandler.java:454)
> solr|   at
> org.apache.solr.client.solrj.io.stream.TupleStream.lambda$writeMap$0(TupleStream.java:84)
> solr|   at
> org.apache.solr.common.util.JsonTextWriter.writeIterator(JsonTextWriter.java:141)
> solr|   at
> org.apache.solr.common.util.TextWriter.writeVal(TextWriter.java:67)
> solr|   at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:152)
> solr|   at
> org.apache.solr.common.util.JsonTextWriter$2.put(JsonTextWriter.java:176)
> solr|   at
> org.apache.solr.client.solrj.io.stream.TupleStream.writeMap(TupleStream.java:81)
> solr|   at
> org.apache.solr.common.util.JsonTextWriter.writeMap(JsonTextWriter.java:164)
> solr|   at
> org.apache.solr.common.util.TextWriter.writeVal(TextWriter.java:69)
> solr|   at
> org.apache.solr.response.TextResponseWriter.writeVal(TextResponseWriter.java:152)
> solr|   at
> org.apache.solr.common.util.JsonTextWriter.writeNamedListAsMapWithDups(JsonTextWriter.java:386)
> solr|   at
> org.apache.solr.common.util.JsonTextWriter.writeNamedList(JsonTextWriter.java:292)
> solr|   at
> org.apache.solr.response.JSONWriter.writeResponse(JSONWriter.java:73)
> solr|   at
> org.apache.solr.response.JSONResponseWriter.write(JSONResponseWriter.java:66)
> solr|   at
> org.apache.solr.response.QueryResponseWriterUtil.writeQueryResponse(QueryResponseWriterUtil.java:65)
> solr|   at
> org.apache.solr.servlet.HttpSolrCall.writeResponse(HttpSolrCall.java:892)
> solr|   at
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:594)
> solr|   at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:419)
> solr|   at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:351)
> solr|   at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
> solr|   at
> org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:311)
> solr|   at
> org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:265)
> solr|   at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
> solr|   at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
> solr|   at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
> solr|   at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
> solr|   at
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
> solr|   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
> solr|   at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1711)
> solr|   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
> solr|   at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1347)
> solr|   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
> solr|   at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
> solr|   at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1678)
> solr|   at
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
> solr|   at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1249)
> solr|   at
> org.eclipse.jetty.server.handler.

Re: Cursor Performance Issue

[Mike Drob]

You should be using docvalues on your id, but note that switching this
would require a reindex.

On Wed, Jan 13, 2021 at 6:04 AM Ajay Sharma 
wrote:

> Hi All,
>
> I have used cursors to search and export documents in solr according to
>
> https://lucene.apache.org/solr/guide/6_6/pagination-of-results.html#fetching-a-large-number-of-sorted-results-cursors
>
> Solr version: 6.5.0
> No of Documents: 10 crore
>
> Before implementing cursor, I was using the start and rows parameter to
> fetch records
> Service response time used to be 2 sec
>
> *Before implementing Cursor Solr URL:*
> http://localhost:8080/solr/search/select?q=bird
> toy&qt=mapping&ps=3&rows=25&mm=100
>
> Request handler Looks like this: fl contains approx 20 fields
> 
> 
> edismax
> on
> 0.01
> 
> 
> id,refid,title,smalldesc:""
> 
>
> none
> json
> 25
> 15000
> smalldesc
> title_text
> titlews^3
> sdescnisq
> 1
> 
> 2<-1 4<70%
> 
> 
>
> Sharing Response with EchoParams=all > Qtime is 6
> responseHeader: {
> status: 0,
> QTime: 6,
> params: {
> ps: "3",
> echoParams: "all",
> indent: "on",
> fl: "id,refid,title,smalldesc:"",
> tie: "0.01",
> defType: "edismax",
> qf: "customphonetic",
> wt: "json",
>qs: "1",
>qt: "mapping",
>rows: "25",
>q: "bird toy",
>timeAllowed: "15000"
> }
> },
> response: {
> numFound: 17,
> start: 0,
> maxScore: 26.616478,
> docs: [
>   {
> id: "22347708097",
> refid: "152585558",
> title: "Round BIRD COLOURFUL SWINGING CIRCULAR SITTING TOY",
> smalldesc: "",
> score: 26.616478
>  }
> ]
> }
>
> I am facing a performance issue now after implementing the cursor. Service
> response time is increased 3 to 4 times .i.e. 8 sec in some cases
>
> *After implementing Cursor query is-*
> localhost:8080/solr/search/select?q=bird
> toy&qt=cursor&ps=3&rows=1000&mm=100&sort=score desc,id asc&cursorMark=*
>
> Just added &sort=score desc,id asc&cursorMark=* to the before query and
> rows to be fetched is 1000 now and fl contains just a single field
>
> Request handler remains same as before just changed the name and made fl
> change and added df in defaults
>
> 
>
>   edismax
>   on
>   0.01
>
>
>   refid
>
>
>   none
>   json
>   1000
>   smalldesc
>   title_text
>   titlews^3
>   sdescnisq
>   1
>   2<-1 4<70%
>   product_titles
>
> 
>
> Response with Cursor and echoParams=all-> *Qtime is now 17* i.e approx 3
> time of previous qtime
> responseHeader: {
> status: 0,
> QTime: 17,
> params: {
> df: "product_titles",
> ps: "3",
> echoParams: "all",
> indent: "on",
> fl: "refid",
> tie: "0.01",
> defType: "edismax",
> qf: "customphonetic",
> qs: "1",
> qt: "cursor",
> sort: "score desc,id asc",
> rows: "1000",
> q: "bird toy",
> cursorMark: "*",
> }
> },
> response: {
> numFound: 17,
> start: 0,
> docs: [
> {
> refid: "152585558"
> },
> {
> refid: "157276077"
> }
> ]
> }
>
>
> When i curl http://localhost:8080/solr/search/select?q=bird
> toy&qt=mapping&ps=3&rows=25&mm=100, i can get results in 3 seconds.
> When i curl localhost:8080/solr/search/select?q=bird
> toy&qt=cursor&ps=3&rows=1000&mm=100&sort=score desc,id asc&cursorMark=* it
> consumed 8 seconds to return result even if the result count=0
>
> BTW, the id schema definition is used in sort
>  omitNorms="true" multiValued="false"/>
>
> Is it due to the sort I have applied or I have implemented it in the wrong
> way?
> Please help or provide the direction to solve this issue
>
>
> Thanks in advance
>
> --
> Thanks & Regards,
> Ajay Sharma
> Product Search
> Indiamart Intermesh Ltd.
>
> --
>
>

Re: Converting a collection name to an alias

[Mike Drob]

I believe you may be able to use that command (or some combination of
create alias commands) to create an alias from A to A, and then in
the future when you want to change it you can have Alias A to collection B
(assuming this is the point of the alias in the first place).

On Thu, Jan 7, 2021 at 1:53 PM ufuk yılmaz 
wrote:

> Hi,
> I’m aware of that API but it doesn’t do what I actually want.
>
> regards
>
> Sent from Mail for Windows 10
>
> From: matthew sporleder
> Sent: 07 January 2021 22:46
> To: solr-user@lucene.apache.org
> Subject: Re: Converting a collection name to an alias
>
> https://lucene.apache.org/solr/guide/8_1/collections-api.html#rename
>
> On Thu, Jan 7, 2021 at 2:07 PM ufuk yılmaz 
> wrote:
> >
> > Hi again,
> >
> > Lets say I have a collection named A.
> > I’m trying to rename it to A_1, then create an alias named A, which
> points to the A_1 collection.
> > Is this possible without deleting and reindexing the collection from
> scratch?
> >
> > Regards,
> > uyilmaz
> >
>
>

Re: SPLITSHARD - data loss of child documents

[Mike Drob]

I was under the impression that split shard doesn’t work with child
documents, if that is missing from the ref guide we should update it

On Thu, Dec 17, 2020 at 4:30 AM Nussbaum, Ronen 
wrote:

> Hi Everyone,
>
> We're using version 8.6.1 with nested documents.
> I used the SPLITSHARD API and after it finished successfully, I've noticed
> the following:
>
>   1.  Most of child documents are missing - before the split: ~600M,
> after: 68M
>   2.  Retrieving a document with its children, shows child documents that
> do not belong to this parent (their parentID value is different than
> parent's ID).
>
> I didn't see any limitation in the API documentation.
> Do you have any suggestions?
>
> Thanks in advance,
> Ronen.
>
>
> This electronic message may contain proprietary and confidential
> information of Verint Systems Inc., its affiliates and/or subsidiaries. The
> information is intended to be for the use of the individual(s) or
> entity(ies) named above. If you are not the intended recipient (or
> authorized to receive this e-mail for the intended recipient), you may not
> use, copy, disclose or distribute to anyone this message or any information
> contained in this message. If you have received this electronic message in
> error, please notify us by replying to this e-mail.
>

Re: solr 8.6.3 and noggit

[Mike Drob]

Noggit code was forked into Solr, see SOLR-13427
https://github.com/apache/lucene-solr/blob/releases/lucene-solr/8.6.3/solr/solrj/src/java/org/noggit/ObjectBuilder.java

It looks like that particular method was added in 8.4 via
https://issues.apache.org/jira/browse/SOLR-13824

Is it possible you're using an older SolrJ against a newer Solr server (or
vice versa).

Mike

On Fri, Nov 20, 2020 at 2:25 PM Susmit Shukla 
wrote:

> Hi,
> got this error using streaming with solrj 8.6.3 . does it use noggit-0.8.
> It was not mentioned in dependencies
> https://github.com/apache/lucene-solr/blob/branch_8_6/solr/solrj/ivy.xml
>
> Caused by: java.lang.NoSuchMethodError: 'java.lang.Object
> org.noggit.ObjectBuilder.getValStrict()'
>
> at org.apache.solr.common.util.Utils.fromJSON(Utils.java:284)
> ~[solr-solrj-8.6.3.jar:8.6.3 e001c2221812a0ba9e9378855040ce72f93eced4 -
> jasongerlowski - 2020-10-03 18:12:06]
>

Re: download binary files will not uncompress

[Mike Drob]

Routing back to the mailing list, please do not reply directly to
individual emails.

You did not download the complete file, the releases should be
approximately 180MB, not the 30KB that you show.

Try downloading from a different mirror, or check if you are behind a proxy
or firewall preventing the downloads.


On Tue, Nov 3, 2020 at 4:51 PM James Rome  wrote:

> jar@jarfx ~/.gnupg $ gpg --import ~/download/KEYS
> gpg: key B83EA82A0AFCEE7C: public key "Yonik Seeley "
> imported
> gpg: key E48025ED13E57FFC: public key "Upayavira "
> imported
>
> ...
>
> jar@jarfx ~/download $ ls -l solr*
> -rw-r--r-- 1 root root 30690 Nov  3 17:00 solr-8.6.3.tgz
> -rw-r--r-- 1 root root   833 Oct  3 21:44 solr-8.6.3.tgz.asc
> -rw-r--r-- 1 root root   145 Oct  3 21:44 solr-8.6.3.tgz.sha512
> -rw-r--r-- 1 root root 30718 Nov  3 17:01 solr-8.6.3.zip
>
> gpg --verify  solr-8.6.3.tgz.asc solr-8.6.3.tgz
> gpg: Signature made Sat 03 Oct 2020 06:17:01 PM EDT
> gpg:using RSA key 902CC51935C140BF820230961FD5295281436075
> gpg: BAD signature from "Jason Gerlowski (CODE SIGNING KEY)
> " [unknown]
>
> jar@jarfx ~/download $ tar xvf solr-8.6.3.tgz
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
>
> James A. Rome
> 116 Claymore Lane
> Oak Ridge, TN 37830-7674
> 865 482-5643; Cell: 865 566-7991
> jamesr...@gmail.com
> https://jamesrome.net
>
> On 11/3/20 5:20 PM, Mike Drob wrote:
> > Can you check the signatures to make sure your downloads were not
> > corrupted? I just checked and was able to download and uncompress both of
> > them.
> >
> > Also, depending on your version of tar, you don't want the - for your
> > flags... tar xf solr-8.6.3.tgz
> >
> > Mike
> >
> > On Tue, Nov 3, 2020 at 4:15 PM James Rome  wrote:
> >
> >> # Source release: solr-8.6.3-src.tgz
> >> <
> >>
> https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3-src.tgz
> >
> >>
> >> [PGP
> >> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3-src.tgz.asc
> >]
> >> [SHA512
> >> <
> https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3-src.tgz.sha512
> >>> ]
> >> # Binary releases: solr-8.6.3.tgz
> >> <https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3.tgz
> >
> >> [PGP
> >> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.tgz.asc>]
> >> [SHA512
> >> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.tgz.sha512>]
> >> / solr-8.6.3.zip
> >> <https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3.zip
> >
> >> [PGP
> >> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.zip.asc>]
> >> [SHA512
> >> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.zip.sha512>]
> >>
> >>unzip solr-8.6.3.zip
> >> Archive:  solr-8.6.3.zip
> >> End-of-central-directory signature not found.  Either this file is
> not
> >> a zipfile, or it constitutes one disk of a multi-part archive. In
> the
> >> latter case the central directory and zipfile comment will be found
> on
> >> the last disk(s) of this archive.
> >>
> >>
> >> and
> >>
> >> # tar -xvf solr-8.6.3.tgz
> >>
> >> gzip: stdin: not in gzip format
> >> tar: Child returned status 1
> >> tar: Error is not recoverable: exiting now
> >>
> >> --
> >> James A. Rome
> >>
> >> https://jamesrome.net
> >>
> >>
>

Re: download binary files will not uncompress

[Mike Drob]

Can you check the signatures to make sure your downloads were not
corrupted? I just checked and was able to download and uncompress both of
them.

Also, depending on your version of tar, you don't want the - for your
flags... tar xf solr-8.6.3.tgz

Mike

On Tue, Nov 3, 2020 at 4:15 PM James Rome  wrote:

> # Source release: solr-8.6.3-src.tgz
> <
> https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3-src.tgz>
>
> [PGP
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3-src.tgz.asc>]
> [SHA512
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3-src.tgz.sha512
> >]
> # Binary releases: solr-8.6.3.tgz
> <https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3.tgz>
> [PGP
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.tgz.asc>]
> [SHA512
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.tgz.sha512>]
> / solr-8.6.3.zip
> <https://www.apache.org/dyn/closer.lua/lucene/solr/8.6.3/solr-8.6.3.zip>
> [PGP
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.zip.asc>]
> [SHA512
> <https://downloads.apache.org/lucene/solr/8.6.3/solr-8.6.3.zip.sha512>]
>
>   unzip solr-8.6.3.zip
> Archive:  solr-8.6.3.zip
>End-of-central-directory signature not found.  Either this file is not
>a zipfile, or it constitutes one disk of a multi-part archive. In the
>latter case the central directory and zipfile comment will be found on
>the last disk(s) of this archive.
>
>
> and
>
> # tar -xvf solr-8.6.3.tgz
>
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> tar: Error is not recoverable: exiting now
>
> --
> James A. Rome
>
> https://jamesrome.net
>
>

Re: Solr dependency update at Apache Beam - which versions should be supported

[Mike Drob]

Piotr,

Based on the questions that we've seen over the past month on this list,
there are still users with Solr on 6, 7, and 8. I suspect there are still
Solr 5 users out there too, although they don't appear to be asking for
help - likely they are in set it and forget it mode.

Solr 7 may not be officially deprecated on our site, but it's pretty old at
this point and we're not doing any development on it outside of mybe a
very high profile security fix. Even then, we might acknowledge it and
recommend users update to 8.x anyway.

The index files generated by Lucene and consumed by Solr are backwards
compatible up to one major version. Some of the API remains compatible, a
client issuing simple queries to Solr 5 would probably work fine even
against Solr 9 when it comes out eventually. A client doing admin
operations will be less certain. I don't know enough about Beam to tell you
where on the spectrum your use will fall.

I'm not sure if this was helpful or not, but maybe it is a nudge in the
right direction.

Good luck,
Mike


On Tue, Oct 27, 2020 at 11:09 AM Piotr Szuberski <
piotr.szuber...@polidea.com> wrote:

> Hi,
>
> We are working on dependency updates at Apache Beam and I would like to
> consult which versions should be supported so we don't break any existing
> users.
>
> Previously the supported Solr version was 5.5.4.
>
> Versions 8.x.y and 7.x.y naturally come to mind as they are the only not
> deprecated. But maybe there are users that use some earlier versions?
>
> Are these versions backwards-compatible or there are things to be aware of?
>
> Regards
>

Re: Folding Repeated Letters

[Mike Drob]

I was thinking about that, but there are words that are legitimately
different with repeated consonants. My primary school teacher lost hair
over getting us to learn the difference between desert and dessert.

Maybe we need something that can borrow the boosting behaviour of fuzzy
query - match the exact term, but also the neighbors with a slight deboost,
so that if the main term exists those others won't show up.

On Thu, Oct 8, 2020 at 5:46 PM Andy Webb  wrote:

> How about something like this?
>
> {
> "add-field-type": [
> {
> "name": "norepeat",
> "class": "solr.TextField",
> "analyzer": {
> "tokenizer": {
> "class": "solr.StandardTokenizerFactory"
> },
> "filters": [
> {
> "class": "solr.LowerCaseFilterFactory"
> },
> {
> "class": "solr.PatternReplaceFilterFactory",
> "pattern": "(.)\\1+",
> "replacement": "$1"
> }
> ]
> }
> }
> ]
> }
>
> This finds a match...
>
> http://localhost:8983/solr/#/norepeat/analysis?analysis.fieldvalue=Yes&analysis.query=YyyeeEssSs&analysis.fieldtype=norepeat
>
> Andy
>
>
>
> On Thu, 8 Oct 2020 at 23:02, Mike Drob  wrote:
>
> > I'm looking for a way to transform words with repeated letters into the
> > same token - does something like this exist out of the box? Do our
> stemmers
> > support it?
> >
> > For example, say I would want all of these terms to return the same
> search
> > results:
> >
> > YES
> > YESSS
> > YYYEEESSS
> > YYEE[...]S
> >
> > I don't know how long a user would hold down the S key at the end to
> > capture their level of excitement, and I don't want to manually define
> > synonyms for every length.
> >
> > I'm pretty sure that I don't want PhoneticFilter here, maybe
> > PatternReplace? Not a huge fan of how that one is configured, and I think
> > I'd have to set up a bunch of patterns inline for it?
> >
> > Mike
> >
>

Folding Repeated Letters

[Mike Drob]

I'm looking for a way to transform words with repeated letters into the
same token - does something like this exist out of the box? Do our stemmers
support it?

For example, say I would want all of these terms to return the same search
results:

YES
YESSS
YYYEEESSS
YYEE[...]S

I don't know how long a user would hold down the S key at the end to
capture their level of excitement, and I don't want to manually define
synonyms for every length.

I'm pretty sure that I don't want PhoneticFilter here, maybe
PatternReplace? Not a huge fan of how that one is configured, and I think
I'd have to set up a bunch of patterns inline for it?

Mike

Re: Term too complex for spellcheck.q param

[Mike Drob]

Right now the only solution is to use a shorter term.

In a fuzzy query you could also try using a lower edit distance e.g. term~1
(default is 2), but I’m not sure what the syntax for a spellcheck would be.

Mike

On Wed, Oct 7, 2020 at 2:59 PM gnandre  wrote:

> Hi,
>
> I am getting following error when I pass '
> 김포오피➬유유닷컴➬✗UUDAT3.COM유유닷컴김포풀싸롱て김포오피ふ김포휴게텔け김포마사지❂김포립카페じ김포안마
> ' in spellcheck.q param. How to avoid this error? I am using Solr 8.5.2
>
> {
>   "error": {
> "code": 500,
> "msg": "Term too complex: 김포오피➬유유닷컴➬✗uudat3.com
> 유유닷컴김포풀싸롱て김포오피ふ김포휴게텔け김포마사지❂김포립카페じ김포안마",
> "trace": "org.apache.lucene.search.FuzzyTermsEnum$FuzzyTermsException:
> Term too complex:
> 김포오피➬유유닷컴➬✗uudat3.com유유닷컴김포풀싸롱て김포오피ふ김포휴게텔け김포마사지❂김포립카페じ김포안마\n\tat
>
> org.apache.lucene.search.FuzzyAutomatonBuilder.buildAutomatonSet(FuzzyAutomatonBuilder.java:63)\n\tat
>
> org.apache.lucene.search.FuzzyTermsEnum$AutomatonAttributeImpl.init(FuzzyTermsEnum.java:365)\n\tat
>
> org.apache.lucene.search.FuzzyTermsEnum.(FuzzyTermsEnum.java:125)\n\tat
>
> org.apache.lucene.search.FuzzyTermsEnum.(FuzzyTermsEnum.java:92)\n\tat
>
> org.apache.lucene.search.spell.DirectSpellChecker.suggestSimilar(DirectSpellChecker.java:425)\n\tat
>
> org.apache.lucene.search.spell.DirectSpellChecker.suggestSimilar(DirectSpellChecker.java:376)\n\tat
>
> org.apache.solr.spelling.DirectSolrSpellChecker.getSuggestions(DirectSolrSpellChecker.java:196)\n\tat
>
> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:195)\n\tat
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:328)\n\tat
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)\n\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2596)\n\tat
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:802)\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:579)\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\n\tat
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)\n\tat
>
> org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
>
> org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:500)\n\tat
>
> org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)\n\tat
> org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)\n\tat
>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)\n\tat
> org.eclipse.jetty.io
> .AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)\n\tat
> org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)\n\tat
> org.eclipse.jetty.io.Ch

RE: Why use a different analyzer for "index" and "query"?

[Dunham-Wilkie, Mike CITZ:EX]

Hi Steven, 

I can think of one case.  If we have an index of database table or column 
names, e.g., words like 'THIS_IS_A_TABLE_NAME', we may want to split the name 
at the underscores when indexing (as well as keep the original), since the 
individual parts might be significant and meaningful.  When querying, though, 
if the searcher types in THIS_IS_A_TABLE_NAME then they are likely looking for 
the whole string, so we wouldn't want to split it apart.

There also seems to be a debate on whether the SYNONYM filter should be 
included on indexing, on querying, or on both.  Google "solr synonyms index vs 
query"

Mike

-Original Message-
From: Steven White  
Sent: September 10, 2020 8:19 AM
To: solr-user@lucene.apache.org
Subject: Why use a different analyzer for "index" and "query"?

[EXTERNAL] This email came from an external source. Only open attachments or 
links that you are expecting from a known sender.


Hi everyone,

In Solr's schema, I have come across field types that use a different logic for 
"index" than for "query".  To be clear, I"m talking about this block:


  
   
  
  
   
  


Why would one want to not use the same logic for both and simply use:


  
   
  


What are real word use cases to use a different analyzer for index and query?

Thanks,

Steve

Lowercase-ing everything but acronyms

[Dunham-Wilkie, Mike CITZ:EX]

Hi SOLR list,

I'm currently using the White Space tokenizer and the Lower Case filter with 
SOLR 7.3.  I'd like to modify the logic to keep any tokens that are entirely 
upper case as upper case, and just apply the Lower Case filter (or something 
equivalent) to the remaining tokens.  Is there a way to do this using 
tokenizers and filters?

Thanks
Mike


Mike Dunham-Wilkie | Senior Spatial Data Administration Analyst | PHONE... 
778-676-1791
Data Systems & Services - Digital Platforms and Data Division - Ministry of 
Citizens' Services

For faster response and/or future inquires, the following email addresses are 
monitored continuously:
BC Geographic Warehouse (BCGW) and Replication/ETL | DataBC Data Architecture 
Services (databc...@gov.bc.ca<mailto:databc...@gov.bc.ca>)
BC Data Catalogue (BCDC) and Open Data | DataBC Catalogue Services 
(data...@gov.bc.ca<mailto:data...@gov.bc.ca>)

Re: Adding solr-core via maven fails

[Mike Drob]

Does it fail similarly on 8.5.0 and .1?

On Thu, Jul 2, 2020 at 6:38 AM Erick Erickson 
wrote:

> There have been some issues with Maven, see:
> https://issues.apache.org/jira/browse/LUCENE-9170
>
> However, we do not officially support Maven builds, they’re there as a
> convenience, so there may still
> be issues in future.
>
> > On Jul 2, 2020, at 1:27 AM, Ali Akhtar  wrote:
> >
> > If I try adding solr-core to an existing project, e.g (SBT):
> >
> > libraryDependencies += "org.apache.solr" % "solr-core" % "8.5.2"
> >
> > It fails due a 404 on the dependencies:
> >
> > Extracting structure failed
> > stack trace is suppressed; run last update for the full output
> > stack trace is suppressed; run last ssExtractDependencies for the full
> > output
> > (update) sbt.librarymanagement.ResolveException: Error downloading
> > org.restlet.jee:org.restlet:2.4.0
> > Not found
> > Not found
> > not found:
> > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > Not found
> > Not found
> > not found:
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> > (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error
> > downloading org.restlet.jee:org.restlet:2.4.0
> > Not found
> > Not found
> > not found:
> > /home/ali/.ivy2/local/org.restlet.jee/org.restlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet/2.4.0/org.restlet-2.4.0.pom
> > Error downloading org.restlet.jee:org.restlet.ext.servlet:2.4.0
> > Not found
> > Not found
> > not found:
> >
> /home/ali/.ivy2/local/org.restlet.jee/org.restlet.ext.servlet/2.4.0/ivys/ivy.xml
> > not found:
> >
> https://repo1.maven.org/maven2/org/restlet/jee/org.restlet.ext.servlet/2.4.0/org.restlet.ext.servlet-2.4.0.pom
> >
> >
> >
> > Any ideas? Do I need to add a specific repository to get it to compile?
>
>

Re: [EXTERNAL] Getting rid of Master/Slave nomenclature in Solr

[Mike Drob]

Brend,

I appreciate that you are trying to examine this issue from multiple sides
and consider future implications, but I don’t think that is a stirring
argument. By analogy, if we are out of eggs and my wife asks me to go to
the store to get some, refusing to do so on the basis that she might call
me while I’m there and also ask me to get milk would not be reasonable.

What will come next may be an interesting question philosophically, but we
are not discussing abstract concepts here. There is a concrete issue
identified, and we’re soliciting input in how best to address it.

Thank you for the suggestion of "guide/follower"

Mike

On Wed, Jun 24, 2020 at 6:30 AM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

> I'm following this thread now for a while and I can understand
> the wish to change some naming/wording/speech in one or the other
> programs but I always get back to the one question:
> "Is it the weapon which kills people or the hand controlled by
> the mind which fires the weapon?"
>
> The thread started with slave - slavery, then turned over to master
> and followed by leader (for me as a german... you know).
> What will come next?
>
> And more over, we now discuss about changes in the source code and
> due to this there need to be changes to the documentation.
> What about the books people wrote about this programs and source code,
> should we force this authors to rewrite their books?
> May be we should file a request to all web search engines to reject
> all stored content about these "banned" words?
> And contact all web hosters about providing bad content.
>
> To sum things up, within my 40 years of computer science and writing
> programs I have never had a nanosecond any thoughts about words
> like master, slave, leader, ... other than thinking about computers
> and programming.
>
> Just my 2 cents.
>
> For what it is worth, I tend to guide/follower if there "must be" any
> changes.
>
> Bernd
>

Re: [EXTERNAL] Re: Getting rid of Master/Slave nomenclature in Solr

[Mike Drob]

I personally think that using Solr cloud terminology for this would be fine
with leader/follower. The leader is the one that accepts updates, followers
cascade the updates somehow. The presence of ZK or election doesn’t really
change this detail.

However, if folks feel that it’s confusing, then I can’t tell them that
they’re not confused. Especially when they’re working with others who have
less Solr experience than we do and are less familiar with the intricacies.

Primary/Replica seems acceptable. Coordinator instead of Overseer seems
acceptable.

Would love to see this in 9.0!

Mike

On Thu, Jun 18, 2020 at 8:25 AM John Gallagher
 wrote:

> While on the topic of renaming roles, I'd like to propose finding a better
> term than "overseer" which has historical slavery connotations as well.
> Director, perhaps?
>
>
> John Gallagher
>
> On Thu, Jun 18, 2020 at 8:48 AM Jason Gerlowski 
> wrote:
>
> > +1 to rename master/slave, and +1 to choosing terminology distinct
> > from what's used for SolrCloud.  I could be happy with several of the
> > proposed options.  Since a good few have been proposed though, maybe
> > an eventual vote thread is the most organized way to aggregate the
> > opinions here.
> >
> > I'm less positive about the prospect of changing the name of our
> > primary git branch.  Most projects that contributors might come from,
> > most tutorials out there to learn git, most tools built on top of git
> > - the majority are going to assume "master" as the main branch.  I
> > appreciate the change that Github is trying to effect in changing the
> > default for new projects, but it'll be a long time before that
> > competes with the huge bulk of projects, documentation, etc. out there
> > using "master".  Our contributors are smart and I'm sure they'd figure
> > it out if we used "main" or something else instead, but having a
> > non-standard git setup would be one more "papercut" in understanding
> > how to contribute to a project that already makes that harder than it
> > should.
> >
> > Jason
> >
> >
> > On Thu, Jun 18, 2020 at 7:33 AM Demian Katz 
> > wrote:
> > >
> > > Regarding people having a problem with the word "master" -- GitHub is
> > changing the default branch name away from "master," even in isolation
> from
> > a "slave" pairing... so the terminology seems to be falling out of favor
> in
> > all contexts. See:
> > >
> > >
> >
> https://www.cnet.com/news/microsofts-github-is-removing-coding-terms-like-master-and-slave/
> > >
> > > I'm not here to start a debate about the semantics of that, just to
> > provide evidence that in some communities, the term "master" is causing
> > concern all by itself. If we're going to make the change anyway, it might
> > be best to get it over with and pick the most appropriate terminology we
> > can agree upon, rather than trying to minimize the amount of change. It's
> > going to be backward breaking anyway, so we might as well do it all now
> > rather than risk having to go through two separate breaking changes at
> > different points in time.
> > >
> > > - Demian
> > >
> > > -Original Message-
> > > From: Noble Paul 
> > > Sent: Thursday, June 18, 2020 1:51 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: [EXTERNAL] Re: Getting rid of Master/Slave nomenclature in
> Solr
> > >
> > > Looking at the code I see a 692 occurrences of the word "slave".
> > > Mostly variable names and ref guide docs.
> > >
> > > The word "slave" is present in the responses as well. Any change in the
> > request param/response payload is backward incompatible.
> > >
> > > I have no objection to changing the names in ref guide and other
> > internal variables. Going ahead with backward incompatible changes is
> > painful. If somebody has the appetite to take it up, it's OK
> > >
> > > If we must change, master/follower can be a good enough option.
> > >
> > > master (noun): A man in charge of an organization or group.
> > > master(adj) : having or showing very great skill or proficiency.
> > > master(verb): acquire complete knowledge or skill in (a subject,
> > technique, or art).
> > > master (verb): gain control of; overcome.
> > >
> > > I hope nobody has a problem with the term "master"
> > >
> > > On Thu, Jun 18, 2020 at 

Re: Master Slave Terminology

[Mike Drob]

Hi Jan,

Can you link to the discussion? I searched the dev list and didn’t see
anything, is it on slack or a jira or somewhere else?

Mike

On Wed, Jun 17, 2020 at 1:51 AM Jan Høydahl  wrote:

> Hi Kaya,
>
> Thanks for bringing it up. The topic is already being discussed by
> developers, so expect to see some change in this area; Not over-night, but
> incremental.
> Also, if you want to lend a helping hand, patches are more than welcome as
> always.
>
> Jan
>
> > 17. jun. 2020 kl. 04:22 skrev Kayak28 :
> >
> > Hello, Community:
> >
> > As the Github and Python will replace terminologies that relative to
> > slavery,
> > why don't we replace master-slave for Solr as well?
> >
> > https://developers.srad.jp/story/18/09/14/0935201/
> >
> https://developer-tech.com/news/2020/jun/15/github-replace-slavery-terms-master-whitelist/
> >
> > --
> >
> > Sincerely,
> > Kaya
> > github: https://github.com/28kayak
>
>

[ANNOUNCE] Apache Solr 8.5.2 released

[Mike Drob]

26 May 2020, Apache Solr™ 8.5.2 available

The Lucene PMC is pleased to announce the release of Apache Solr 8.5.2

Solr is the popular, blazing fast, open source NoSQL search platform from
the Apache Lucene project. Its major features include powerful full-text
search, hit highlighting, faceted search and analytics, rich document
parsing, geospatial search, extensive REST APIs as well as parallel SQL.
Solr is enterprise grade, secure and highly scalable, providing fault
tolerant distributed search and indexing, and powers the search and
navigation features of many of the world's largest internet sites.

This release contains two bug fixes. The release is available for immediate
download at:

The release is available for immediate download at:

https://lucene.apache.org/solr/downloads.html

Solr 8.5.2 Bug Fixes:

   - SOLR-14411 : Fix
   regression from SOLR-14359 (Admin UI 'Select an Option')
   - SOLR-14471 : base
   replica selection strategy not applied to "last place" shards.preference
   matches

Solr 8.5.2 also includes 1 bugfix in the corresponding Apache Lucene
release:



Please report any feedback to the mailing lists (
https://lucene.apache.org/solr/community.html#mailing-lists-irc)

Note: The Apache Software Foundation uses an extensive mirroring network
for distributing releases. It is possible that the mirror you are using may
not have replicated the release yet. If that is the case, please try
another mirror. This also goes for Maven access.

Re: Solr 8.5.1 startup error - lengthTag=109, too big.

[Mike Drob]

Did you have SSL enabled with 8.2.1?

The error looks common to certificate handling and not specific to Solr.

I would verify that you have no extra characters in your certificate file
(including line endings) and that the keystore type that you specified
matches the file you are presenting (JKS or PKCS12)

Mike

On Sat, May 23, 2020 at 10:11 PM Zheng Lin Edwin Yeo 
wrote:

> Hi,
>
> I'm trying to upgrade from Solr 8.2.1 to Solr 8.5.1, with Solr SSL
> Authentication and Authorization.
>
> However, I get the following error when I enable SSL. The Solr itself can
> start up if there is no SSL.  The main error that I see is this
>
>   java.io.IOException: DerInputStream.getLength(): lengthTag=109, too big.
>
> What could be the reason that causes this?
>
>
> INFO  - 2020-05-24 10:38:20.080;
> org.apache.solr.util.configuration.SSLConfigurations; Setting
> javax.net.ssl.keyStorePassword
> INFO  - 2020-05-24 10:38:20.081;
> org.apache.solr.util.configuration.SSLConfigurations; Setting
> javax.net.ssl.trustStorePassword
> Waiting up to 120 to see Solr running on port 8983
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
> at org.eclipse.jetty.start.Main.start(Main.java:491)
> at org.eclipse.jetty.start.Main.main(Main.java:77)d
> Caused by: java.security.PrivilegedActionException: java.io.IOException:
> DerInputStream.getLength(): lengthTag=109, too big.
> at java.security.AccessController.doPrivileged(Native Method)
> at
> org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1837)
> ... 7 more
> Caused by: java.io.IOException: DerInputStream.getLength(): lengthTag=109,
> too big.
> at sun.security.util.DerInputStream.getLength(Unknown Source)
> at sun.security.util.DerValue.init(Unknown Source)
> at sun.security.util.DerValue.(Unknown Source)
> at sun.security.util.DerValue.(Unknown Source)
> at sun.security.pkcs12.PKCS12KeyStore.engineLoad(Unknown Source)
> at java.security.KeyStore.load(Unknown Source)
> at
>
> org.eclipse.jetty.util.security.CertificateUtils.getKeyStore(CertificateUtils.java:54)
> at
>
> org.eclipse.jetty.util.ssl.SslContextFactory.loadKeyStore(SslContextFactory.java:1188)
> at
>
> org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:323)
> at
>
> org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
>
> org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:92)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169)
> at
>
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117)
> at
>
> org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:320)
> at
>
> org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81)
> at
> org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:231)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at org.eclipse.jetty.server.Server.doStart(Server.java:385)
> at
>
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72)
> at
>
> org.eclipse.jetty.xml.XmlConfiguration.lambda$main$0(XmlConfiguration.java:1888)
> ... 9 more
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
> at java.lang.reflect.Method.invoke(Unknown Source)
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:218)
> at org.eclipse.jetty.start.Main.start(Main.java:491)
> at org.eclipse.jetty.

Re: Download a pre-release version? 8.6

[Mike Drob]

We could theoretically include this in a 8.5.2 version which should be
released soon. The change looks minimally risky to backport?

On Fri, May 15, 2020 at 3:43 PM Jan Høydahl  wrote:

> Check Jenkins:
> https://builds.apache.org/view/L/view/Lucene/job/Solr-Artifacts-8.x/lastSuccessfulBuild/artifact/solr/package/
>
> Jan Høydahl
>
> > 15. mai 2020 kl. 22:27 skrev Phill Campbell
> :
> >
> > Is there a way to download a tgz of the binary of a nightly build or
> similar?
> >
> > I have been testing 8.5.1 and ran into the bug with load balancing.
> > https://issues.apache.org/jira/browse/SOLR-14471 <
> https://issues.apache.org/jira/browse/SOLR-14471>
> >
> > It is a deal breaker for me to move forward with an upgrade of the
> system.
> >
> > I would like to start evaluating a version that has the fix.
> >
> > Is there a place to get a build?
> >
> > Thank you.
>

Re: Possible issue with Stemming and nouns ended with suffix 'ion'

[Mike Drob]

This is how things get stemmed *now*, but I believe there is an open
question as to whether that is how they *should* be stemmed. Specifically,
the case appears to be -ify words not stemming to the same as -ification -
this applies to much more than identify/identification. Also, justify,
fortify, notify, many many others.

$ grep ification /usr/share/dict/words | wc -l
 328

I am by no means an expert on stemming, and if the folks at snowball decide
to tell us that this change is bad or hard because it would overstem some
other words, then I'll happily accept that. But I definitely want to use
their expertise rather than relying on my own.

Mike

On Fri, May 1, 2020 at 10:35 AM Audrey Lorberfeld -
audrey.lorberf...@ibm.com  wrote:

> Unless I'm misunderstanding the bug in question, there is no bug. What you
> are observing is simply just how things get stemmed...
>
> Best,
> Audrey
>
> On 4/30/20, 6:37 PM, "Jhonny Lopez" 
> wrote:
>
> Yes, sounds like worth it.
>
> Thanks guys!
>
> -Original Message-
> From: Mike Drob 
> Sent: jueves, 30 de abril de 2020 5:30 p. m.
> To: solr-user@lucene.apache.org
> Subject: Re: Possible issue with Stemming and nouns ended with suffix
> 'ion'
>
> This email has been sent from a source external to Publicis Groupe.
> Please use caution when clicking links or opening attachments.
> Cet email a été envoyé depuis une source externe à Publicis Groupe.
> Veuillez faire preuve de prudence lorsque vous cliquez sur des liens ou
> lorsque vous ouvrez des pièces jointes.
>
>
>
> Is this worth filing a bug/suggestion to the folks over at
> snowballstem.org?
>
> On Thu, Apr 30, 2020 at 4:08 PM Audrey Lorberfeld -
> audrey.lorberf...@ibm.com  wrote:
>
> > I agree with Erick. I think that's just how the cookie crumbles when
> > stemming. If you have some time on your hands, you can integrate
> > OpenNLP with your Solr instance and start using the lemmas of tokens
> > instead of the stems. In this case, I believe if you were to
> lemmatize
> > both "identify" and "identification," they would both condense to
> "identify."
> >
> > Best,
> > Audrey
> >
> > On 4/30/20, 3:54 PM, "Erick Erickson" 
> wrote:
> >
> > They are being stemmed to two different tokens, “identif” and
> > “identifi”. Stemming is algorithmic and imperfect and in this case
> > you’re getting bitten by that algorithm. It looks like you’re using
> > PorterStemFilter, if you want you can look up the exact algorithm,
> but
> > I don’t think it’s a bug, just one of those little joys of English...
> >
> > To get a clearer picture of exactly what’s being searched, try
> > adding &debug=query to your query, in particular looking at the
> parsed
> > query that’s returned. That’ll tell you a bunch. In this particular
> > case I don’t think it’ll tell you anything more, but for future…
> >
> > Best,
> > Erick
> >
> > On, and un-checking the ‘verbose’ box on the analysis page
> removes
> > a lot of distraction, the detailed information is often TMI ;)
> >
> > > On Apr 30, 2020, at 2:51 PM, Jhonny Lopez <
> > jhonny.lo...@publicismedia.com> wrote:
> > >
> > > Sure, rewriting the message with links for images:
> > >
> > >
> > > We’re facing an issue with stemming in solr. Most of the cases
> > are working correctly, for example, if we search for bidding, solr
> > brings results for bidding, bid, bids, etc. However, with nouns
> ended with ‘ion’
> > suffix, stemming is not working. Even when analyzers seems to have
> > correct stemming of the word, the results are not reflecting that.
> One
> > example. If I search ‘identifying’, this is the output:
> > >
> > > Analyzer (image link):
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd4-2DCp40Cmc0QioS0A-3Fe-3D1f3GJp&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=U-Wmu118X5bfNxDnADO_6ompf9kUxZYHj1DZM2lG4jo&e=
> > >
> > > A clip of results:
> > > "haschildren_b":false,
> > >"isbucket_text_s":"0",
> > >"sectionbody_t":&q

Re: Possible issue with Stemming and nouns ended with suffix 'ion'

[Mike Drob]

Jhonny,

Are you planning on reporting the issue to snowball, or would you prefer
one of us take care of it?
If you do report it, please share the link to the issue or mail archive
back here so that we know when it is resolved and can update our
dependencies.

Thanks,
Mike

On Thu, Apr 30, 2020 at 5:37 PM Jhonny Lopez 
wrote:

> Yes, sounds like worth it.
>
> Thanks guys!
>
> -Original Message-----
> From: Mike Drob 
> Sent: jueves, 30 de abril de 2020 5:30 p. m.
> To: solr-user@lucene.apache.org
> Subject: Re: Possible issue with Stemming and nouns ended with suffix 'ion'
>
> This email has been sent from a source external to Publicis Groupe. Please
> use caution when clicking links or opening attachments.
> Cet email a été envoyé depuis une source externe à Publicis Groupe.
> Veuillez faire preuve de prudence lorsque vous cliquez sur des liens ou
> lorsque vous ouvrez des pièces jointes.
>
>
>
> Is this worth filing a bug/suggestion to the folks over at
> snowballstem.org?
>
> On Thu, Apr 30, 2020 at 4:08 PM Audrey Lorberfeld -
> audrey.lorberf...@ibm.com  wrote:
>
> > I agree with Erick. I think that's just how the cookie crumbles when
> > stemming. If you have some time on your hands, you can integrate
> > OpenNLP with your Solr instance and start using the lemmas of tokens
> > instead of the stems. In this case, I believe if you were to lemmatize
> > both "identify" and "identification," they would both condense to
> "identify."
> >
> > Best,
> > Audrey
> >
> > On 4/30/20, 3:54 PM, "Erick Erickson"  wrote:
> >
> > They are being stemmed to two different tokens, “identif” and
> > “identifi”. Stemming is algorithmic and imperfect and in this case
> > you’re getting bitten by that algorithm. It looks like you’re using
> > PorterStemFilter, if you want you can look up the exact algorithm, but
> > I don’t think it’s a bug, just one of those little joys of English...
> >
> > To get a clearer picture of exactly what’s being searched, try
> > adding &debug=query to your query, in particular looking at the parsed
> > query that’s returned. That’ll tell you a bunch. In this particular
> > case I don’t think it’ll tell you anything more, but for future…
> >
> > Best,
> > Erick
> >
> > On, and un-checking the ‘verbose’ box on the analysis page removes
> > a lot of distraction, the detailed information is often TMI ;)
> >
> > > On Apr 30, 2020, at 2:51 PM, Jhonny Lopez <
> > jhonny.lo...@publicismedia.com> wrote:
> > >
> > > Sure, rewriting the message with links for images:
> > >
> > >
> > > We’re facing an issue with stemming in solr. Most of the cases
> > are working correctly, for example, if we search for bidding, solr
> > brings results for bidding, bid, bids, etc. However, with nouns ended
> with ‘ion’
> > suffix, stemming is not working. Even when analyzers seems to have
> > correct stemming of the word, the results are not reflecting that. One
> > example. If I search ‘identifying’, this is the output:
> > >
> > > Analyzer (image link):
> > >
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd4-2DCp40Cmc0QioS0A-3Fe-3D1f3GJp&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=U-Wmu118X5bfNxDnADO_6ompf9kUxZYHj1DZM2lG4jo&e=
> > >
> > > A clip of results:
> > > "haschildren_b":false,
> > >"isbucket_text_s":"0",
> > >"sectionbody_t":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > >"parsedupdatedby_s":"sitecorecarvaini",
> > >"sectionbody_t_en":"\n\n\nIn order to identify 1st price
> > auctions, leverage the proprietary tools available or manually pull a
> > log file report to understand the trends and gauge auction spread
> > overtime to assess the impact of variable auction
> dynamics.\n\n\n\n\n\n\n",
> > >"hide_section_b":false
> > >
> > >
> > > As you can see, it has used the stemming correctly and brings
> > results for other words based in th

Re: Possible issue with Stemming and nouns ended with suffix 'ion'

[Mike Drob]

Is this worth filing a bug/suggestion to the folks over at snowballstem.org?

On Thu, Apr 30, 2020 at 4:08 PM Audrey Lorberfeld -
audrey.lorberf...@ibm.com  wrote:

> I agree with Erick. I think that's just how the cookie crumbles when
> stemming. If you have some time on your hands, you can integrate OpenNLP
> with your Solr instance and start using the lemmas of tokens instead of the
> stems. In this case, I believe if you were to lemmatize both "identify" and
> "identification," they would both condense to "identify."
>
> Best,
> Audrey
>
> On 4/30/20, 3:54 PM, "Erick Erickson"  wrote:
>
> They are being stemmed to two different tokens, “identif” and
> “identifi”. Stemming is algorithmic and imperfect and in this case you’re
> getting bitten by that algorithm. It looks like you’re using
> PorterStemFilter, if you want you can look up the exact algorithm, but I
> don’t think it’s a bug, just one of those little joys of English...
>
> To get a clearer picture of exactly what’s being searched, try adding
> &debug=query to your query, in particular looking at the parsed query
> that’s returned. That’ll tell you a bunch. In this particular case I don’t
> think it’ll tell you anything more, but for future…
>
> Best,
> Erick
>
> On, and un-checking the ‘verbose’ box on the analysis page removes a
> lot of distraction, the detailed information is often TMI ;)
>
> > On Apr 30, 2020, at 2:51 PM, Jhonny Lopez <
> jhonny.lo...@publicismedia.com> wrote:
> >
> > Sure, rewriting the message with links for images:
> >
> >
> > We’re facing an issue with stemming in solr. Most of the cases are
> working correctly, for example, if we search for bidding, solr brings
> results for bidding, bid, bids, etc. However, with nouns ended with ‘ion’
> suffix, stemming is not working. Even when analyzers seems to have correct
> stemming of the word, the results are not reflecting that. One example. If
> I search ‘identifying’, this is the output:
> >
> > Analyzer (image link):
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd4-2DCp40Cmc0QioS0A-3Fe-3D1f3GJp&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=U-Wmu118X5bfNxDnADO_6ompf9kUxZYHj1DZM2lG4jo&e=
> >
> > A clip of results:
> > "haschildren_b":false,
> >"isbucket_text_s":"0",
> >"sectionbody_t":"\n\n\nIn order to identify 1st price
> auctions, leverage the proprietary tools available or manually pull a log
> file report to understand the trends and gauge auction spread overtime to
> assess the impact of variable auction dynamics.\n\n\n\n\n\n\n",
> >"parsedupdatedby_s":"sitecorecarvaini",
> >"sectionbody_t_en":"\n\n\nIn order to identify 1st price
> auctions, leverage the proprietary tools available or manually pull a log
> file report to understand the trends and gauge auction spread overtime to
> assess the impact of variable auction dynamics.\n\n\n\n\n\n\n",
> >"hide_section_b":false
> >
> >
> > As you can see, it has used the stemming correctly and brings
> results for other words based in the root, in this case “Identify”.
> >
> > However, if I search for “Identification”, this is the output:
> >
> > Analyzer (imagelink):
> >
> https://urldefense.proofpoint.com/v2/url?u=https-3A__1drv.ms_u_s-21AlRTlFq8tQbShd49RpiQObzMgSjVhA&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=_8ViuZIeSRdQjONA8yHWPZIBlhj291HU3JpNIx5a55M&m=8Xt1N2A4ODj--DlLb242c8JMnJr6nWQIwcKjiDiA__s&s=5RlkLH-90sYc4nyIgnPO9MsBlyh7iWSOphEVdjUvTIE&e=
> >
> >
> > Even with proper stemming, solr is only bringing results for the
> word identification (or identifications) but nothing else.
> >
> > The queries are over the same field that has the Porter Stemming
> Filter applied for both, query and index. This behavior is consistent with
> other ‘ion’ ended nouns: representation, modification, etc.
> >
> > Solr Version: 8.1. Does anyone know why is it happening? Is it a bug?
> >
> > Thanks.
> >
> >
> >
> >
> >
> > -Original Message-
> >
> > From: Erick Erickson 
> >
> > Sent: jueves, 30 de abril de 2020 1:47 p. m.
> >
> > To: solr-user@lucene.apache.org
> >
> > Subject: Re: Possible issue with Stemming and nouns ended with
> suffix 'ion'
> >
> >
> >
> > This email has been sent from a source external to Publicis Groupe.
> Please use caution when clicking links or opening attachments.
> >
> > Cet email a été envoyé depuis une source externe à Publicis Groupe.
> Veuillez faire preuve de prudence lorsque vous cliquez sur des liens ou
> lorsque vous ouvrez des pièces jointes.
> >
> >
> >
> >
> >
> >
> >
> > The mail server is pretty aggressive about stripping links, so we
> can’t see the images.
> >
> >
> 

Re: Fuzzy search not working

[Mike Drob]

Pradeep,

First, some background on fuzzy term expansions:

1) A query for foobar~2 is really a query for (foobar OR foobar~1 OR
foobar~2)
2) Fuzzy term expansion will only take the first 50 terms found in the
index and drop the rest.

For implementation notes, see this comment -
https://github.com/apache/lucene-solr/blob/master/lucene/core/src/java/org/apache/lucene/search/FuzzyTermsEnum.java#L229-L232

So in your first search, your available terms in the title_txt_en field are
few enough that "probl~2" does match "problem"
In your second search, with the copy field, there are likely many more
terms in all_text_txt_enus. Here, the edit distance 1 terms crowd out the
edit distance 2 terms and they never match.
You can imagine that the term expands into… "probl OR prob OR probe OR
prob1 OR…"

I don't see a way to specify the number of expansions from a Solr query,
maybe somebody else on the list would know.

But at the end of the day, like Wunder said, you might want a prefix query
based on what you're describing.

Mike


On Mon, Apr 13, 2020 at 6:01 PM Deepu  wrote:

> Corrected Typo mistake.
>
> Hi Team,
>
> We have 8 text fields (*_txt_en) in schema and one multi valued text field
> which is copy field of other text fields, like below.
>
> tittle_txt_en, configuration_summary_txt_en, all_text_txt_ens (multi value
> field)
>
> Observed one issue with Fuzzy match, same term with distance of two(~2) is
> working on individual fields but not returning any results from multi
> valued field.
>
> Term we used is "probl" and document has "problem" term in two text fields,
> so all_text field has two occurrences of 'problem" terms.
>
>
>
> title_txt_en:probl~2. (given results)
>
> all_text_txt_ens:probl~2 (no results)
>
>
>
> is there any other factors involved in distance calculation other
> than Damerau-Levenshtein Distance algoritham?
>
> what might be the reason same input with same distance worked with one
> field and failed with other field in same collection?
>
> is there a way we can get actual distance solr calculated w.r.t specific
> document and specific field ?
>
>
>
> Thanks in advance !!
>
>
> Thanks,
>
> Pradeep
>
> On Mon, Apr 13, 2020 at 2:35 PM Deepu  wrote:
>
> > Hi Team,
> >
> > We have 8 text fields (*_txt_en) in schema and one multi valued text
> field
> > which is copy field of other text fields, like below.
> >
> > tittle_txt_en, configuration_summary_txt_en, all_text_txt_ens (multi
> value
> > field)
> >
> > Observed one issue with Fuzzy match, same term with distance of two(~2)
> is
> > working on individual fields but not returning any results from multi
> > valued field.
> >
> > Term we used is "prob" and document has "problem" term in two text
> fields,
> > so all_text field has two occurrences of 'problem" terms.
> >
> >
> >
> > title_txt_en:prob~2. (given results)
> >
> > all_text_txt_ens:prob~2 (no results)
> >
> >
> >
> > is there any other factors involved in distance calculation other
> > than Damerau-Levenshtein Distance algoritham?
> >
> > what might be the reason same input with same distance worked with one
> > field and failed with other field in same collection?
> >
> > is there a way we can get actual distance solr calculated w.r.t specific
> > document and specific field ?
> >
> >
> >
> > Thanks in advance !!
> >
> >
> > Thanks,
> >
> > Pradeep
> >
>

Re: Apache Solr 8.4.1 Basic Authentication

[Mike Phillips]

The line webResource = client.resource(resourceUrl); defines what action 
I am performing example 
"https://localhost:8985/solr/CoreName/select?q=*%3A*";
Are you setting up your URL correctly. My snippet was outlining the 
additional Authorization header that needs to also be part of the 
request but assuming you were already going to a valid URL.


On 3/26/2020 3:59 PM, Altamirano, Emmanuel wrote:


Thank you so much for replying my email Mike.

I did use now the base64 to encode user and password but now Solr 
doesn’t undertint the credentials:


{Accept=[application/json], Content-Type=[application/json], 
*Authorization*=[Basic c29scjpTb2xyUm9ja3M=]}>] 
ERROR[org.springframework.web.client.HttpClientErrorException: 404 Not 
Found]


Before I got:

{Accept=[application/json], Content-Type=[application/json], 
*Authorization*=[Basic solr:SolrRocks]}>] 
ERROR[org.springframework.web.client.HttpClientErrorException: 401 
Invalid authentication token]


Is there something else that I need to configure?

*Emmanuel Altamirano,*

Consultant- Global Technology

International Operations

*Telephone:*312-985-3149

*Mobile:*312-860-3774

*cid:image001.png@01D02A68.19FA64F0*

555 W. Adams 5^th Floor

Chicago, IL 60661

_transunion.com <http://www.transunion.com/>___

This email including, without limitation, the attachments, if any, 
accompanying this email, may contain information which is confidential 
or privileged and exempt from disclosure under applicable law. The 
information is for the use of the intended recipient. If you are not 
the intended recipient, be aware that any disclosure, copying, 
distribution, review or use of the contents of this email, and/or its 
attachments, is without authorization and is prohibited. If you have 
received this email in error, please notify us by reply email 
immediately and destroy all copies of this email and its attachments.


*From:* Mike Phillips 
*Sent:* Thursday, March 26, 2020 3:10 PM
*To:* Altamirano, Emmanuel 
*Subject:* Re: Apache Solr 8.4.1 Basic Authentication

*EXTERNAL SENDER:* Exercise caution with links and attachments.

I use Jersey to talk to solr. Here is a code snippet. You seem to be 
on the right track but you need to base64 encode the username/password 
bytes.


    String combined = username + ":" + password;
    String  encoded = base64.encode(combined.getBytes());
    String  authHeader = "Basic " + encoded;

    // Setup need to encode the query
    webResource = client.resource(resourceUrl);
    webResource.accept("*.*");

    // Perform request
    response = webResource.header("Content-Type", "application/json")
    .header("Authorization", authHeader)
    .get(ClientResponse.class);
    respStatus = response.getStatus();

On 3/26/2020 12:27 PM, Altamirano, Emmanuel wrote:

Hello everyone,

We recently enable Solr Basic Authentication in our Dev
environment and we are testing Solr security. We followed the
instructions provided in the Apache Solr website and it is working
using curl command.

If you could provide us any advice of how do we need to send the
credentials in the HTTP headers in a Java program? It is very
appreciate it.

HttpHeaders headers= *new*HttpHeaders();

headers.setAccept(Arrays./asList/(MediaType.*/APPLICATION_JSON/*));

headers.setContentType(MediaType.*/APPLICATION_JSON/*);

headers.add("Authorization", "Basic "+ "solr:SolrRocks");

Thanks,

*Emmanuel Altamirano,*

Consultant- Global Technology

International Operations

*Telephone:*312-985-3149

*Mobile:*312-860-3774

*cid:image001.png@01D02A68.19FA64F0*

555 W. Adams 5^th Floor

Chicago, IL 60661

_transunion.com <http://www.transunion.com/>_

This email including, without limitation, the attachments, if any,
accompanying this email, may contain information which is
confidential or privileged and exempt from disclosure under
applicable law. The information is for the use of the intended
recipient. If you are not the intended recipient, be aware that
any disclosure, copying, distribution, review or use of the
contents of this email, and/or its attachments, is without
authorization and is prohibited. If you have received this email
in error, please notify us by reply email immediately and destroy
all copies of this email and its attachments.

SolrCloud location for solr.xml

[Mike Drob]

Hi Searchers!

I was recently looking at some of the start-up logic for Solr and was
interested in cleaning it up a little bit. However, I'm not sure how common
certain deployment scenarios are. Specifically is anybody doing the
following combination:

* Using SolrCloud (i.e. state stored in zookeeper)
* Loading solr.xml from a local solr home rather than zookeeper

Much appreciated! Thanks,
Mike

Re: Is this a bug? Wildcard with PatternReplaceFilterFactory

[Mike Phillips]

It looks like the debug result you are showing me is the results for 
Rod's not Rod’s, but in answer to your question


This is why I think    "Rod’s  finds fields Rod's and 
Rod’s that are now in the index as rod's"


The analysis page shows Rod’s gets stored in the index as:
rod's rods rod s

Field Value (Index)

Rod’s

Analyse Fieldname / FieldType: _text_ Schema Browser 
<https://centos1:8985/solr/#/rat_11/schema?field=_text_>


 *
   Verbose Output

WT

text
raw_bytes
start
end
positionLength
type
termFrequency
position


Rod’s
[52 6f 64 e2 80 99 73]
0
5
1
word
1
1

SF

text
raw_bytes
start
end
positionLength
type
termFrequency
position


Rod’s
[52 6f 64 e2 80 99 73]
0
5
1
word
1
1

WDGF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod’s
[52 6f 64 e2 80 99 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

FGF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod’s
[52 6f 64 e2 80 99 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

PRF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod’s
[52 6f 64 e2 80 99 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

PRF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod's
[52 6f 64 27 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

PRF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod's
[52 6f 64 27 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

PRF

text
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


Rod's
[52 6f 64 27 73]
0
5
2
word
1
1
false


Rods
[52 6f 64 73]
0
5
2
word
1
1
false


Rod
[52 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false

LCF

tex

t
raw_bytes
start
end
positionLength
type
termFrequency
position
keyword


rod's
[72 6f 64 27 73]
0
5
2
word
1
1
false


rods
[72 6f 64 73]
0
5
2
word
1
1
false


rod
[72 6f 64]
0
3
1
word
1
1
false


s
[73]
4
5
1
word
1
2
false



This is  what we were trying to achieve with the class="solr.PatternReplaceFilterFactory" pattern="’" replacement="'"/>



The problem is when using wildcard *Rod’s* we get no hits
||

|"responseHeader":{ "status":0, "QTime":2, "params":{ "q":"*Rod’s*", 
"debugQuery":"on", "_":"1582315262594"}}, 
"response":{"numFound":0,"start":0,"docs":[] }, "debug":{ 
"rawquerystring":"*Rod’s*", "querystring":"*Rod’s*", 
"parsedquery":"_text_:*rod’s*", "parsedquery_toString":"_text_:*rod’s*", 
"explain":{}, "QParser":"LuceneQParser", ... |







On 2/21/2020 11:52 AM, Erick Erickson wrote:

Why do you say “…that are now in the index as rod’s”? You have 
WordDelimiterGraphFilterFactory, which breaks things up. When I put your field 
definition in the schema and use the analysis page, turns “rod’s” into  the 
following 4 tokens:

rod’s
rods
rod
s

And querying on field:”*Rod’s*” works just fine. I’m using 8.x, and when I add 
“&debug=query” to the URL, I see:
{
"responseHeader": {
"status": 0, "QTime": 10, "params": {
"q": "eoe:\"*Rod's*\"", "debug": "query"
}
}, "response": {
"numFound": 1, "start": 0, "docs": [
{
"id": "1", "eoe": "Rod's", "_version_": 1659176849231577088
}
]
}, "debug": {
"rawquerystring": "eoe:\"*Rod's*\"", "querystring": "eoe:\"*Rod's*\"", "parsedquery": "SynonymQuery(Synonym(eoe:*rod's* 
eoe:rod))", "parsedquery_toString": "Synonym(eoe:*rod's* eoe:rod)", "QParser": "LuceneQParser"
}
}

What do you see?

Best,
Erick


On Feb 21, 2020, at 12:57 PM, Mike Phillips  
wrote:

Rod’s  finds fields Rod's and Rod’s that are now in the index as rod's

but *Rod’s* finds nothing because the index now only contains rod's

Is this a bug? Wildcard with PatternReplaceFilterFactory

[Mike Phillips]

Is this a bug? Wildcard with PatternReplaceFilterFactory

Attempting to normalize left and right single and double quotes for searches

‘   Left single quotation mark    '    Single quote
’   Right single quotation mark   '    Single quote
“   Left double quotation mark    "    Double quotes
”   Right double quotation mark   "    Double quotes


    positionIncrementGap="100" multiValued="true">

  
    
    words="stopwords.txt" />
        preserveOriginal="1" catenateWords="1"/>
         
        replacement="'"/>
        replacement="'"/>
    replacement="""/>
    replacement="""/>

    
  
  
    
        preserveOriginal="1" catenateWords="1"/>
    words="stopwords.txt" />
    synonyms="synonyms.txt" ignoreCase="true" expand="true"/>
        replacement="'"/>
        replacement="'"/>
    replacement="""/>
    replacement="""/>

    
  
    

The wildcard seems to NOT utilize the PatternReplaceFilterFactory

Rod’s  finds fields Rod's and Rod’s that are now in the index as rod's

but *Rod’s* finds nothing because the index now only contains rod's

Re: Outdated information on JVM heap sizes in Solr 8.3 documentation?

[Mike Drob]

Erick,

Can you drop a link to that Jira here after you create it?

Many thanks,
Mike

On Fri, Feb 14, 2020 at 6:05 PM Erick Erickson 
wrote:

> I just read that page over and it looks way out of date. I’ll raise
> a JIRA.
>
> > On Feb 14, 2020, at 2:55 PM, Walter Underwood 
> wrote:
> >
> > Yeah, that is pretty outdated. At Netflix, I was running an 8 GB heap
> with Solr 1.3. :-)
> >
> > Every GC I know about has a stop-the-world collector as a last ditch
> measure.
> >
> > G1GC limits the time that the world will stop. It gives up after
> MaxGCPauseMillis
> > milliseconds and leaves the rest of the garbage uncollected. If it has 5
> seconds
> > worth of work to do that, it might take 10 seconds, but in 200 ms
> chunks. It does
> > a lot of other stuff outside of the pauses to make the major collections
> more effective.
> >
> > We wrote Ultraseek in Python+C because Python used reference counting and
> > did not do garbage collection. That is the only way to have no pauses
> with
> > automatic memory management.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >> On Feb 14, 2020, at 11:35 AM, Tom Burton-West 
> wrote:
> >>
> >> Hello,
> >>
> >> In the section on JVM tuning in the  Solr 8.3 documentation (
> >> https://lucene.apache.org/solr/guide/8_3/jvm-settings.html#jvm-settings
> )
> >> there is a paragraph which cautions about setting heap sizes over 2 GB:
> >>
> >> "The larger the heap the longer it takes to do garbage collection. This
> can
> >> mean minor, random pauses or, in extreme cases, "freeze the world"
> pauses
> >> of a minute or more. As a practical matter, this can become a serious
> >> problem for heap sizes that exceed about **two gigabytes**, even if far
> >> more physical memory is available. On robust hardware, you may get
> better
> >> results running multiple JVMs, rather than just one with a large memory
> >> heap. "  (** added by me)
> >>
> >> I suspect this paragraph is severely outdated, but am not a Java expert.
> >> It seems to be contradicted by the statement in "
> >>
> https://lucene.apache.org/solr/guide/8_3/taking-solr-to-production.html#memory-and-gc-settings
> "
> >> "...values between 10 and 20 gigabytes are not uncommon for production
> >> servers"
> >>
> >> Are "freeze the world" pauses still an issue with modern JVM's?
> >> Is it still advisable to avoid heap sizes over 2GB?
> >>
> >> Tom
> >> https://www.hathitrust.org/blogslarge-scale-search
> >
>
>

Re: Modify partial configsets using API

[Mike Drob]

On 2019/05/08 16:52:52, Shawn Heisey  wrote: 
> On 5/8/2019 10:50 AM, Mike Drob wrote:
> > Solr Experts,
> > 
> > Is there an existing API to modify just part of my configset, for example
> > synonyms or stopwords? I see that there is the schema API, but that is
> > pretty specific in scope.
> > 
> > Not sure if I should be looking at configset API to upload a zip with a
> > single file, or if there are more granular options available.
> 
> Here's a documentation link for managed resources:
> 
> https://lucene.apache.org/solr/guide/6_6/managed-resources.html
> 
> That's the 6.6 version of the documentation.  If you're running 
> something newer, which seems likely since 6.6 is quite old now, you 
> might want to look into a later documentation version.
> 
> Thanks,
> Shawn
> 

Thanks Shawn, this looks like it will fit the bill nicely!

One more question that I don't see covered in the documentation - if I have 
multiple collections sharing the same config set, does updating the managed 
stop words for one collection apply the change to all? Is this change persisted 
in zookeeper?

Mike

Modify partial configsets using API

[Mike Drob]

Solr Experts,

Is there an existing API to modify just part of my configset, for example
synonyms or stopwords? I see that there is the schema API, but that is
pretty specific in scope.

Not sure if I should be looking at configset API to upload a zip with a
single file, or if there are more granular options available.

Thanks,
Mike

Highlighting

[Mike Phillips]

I don't understand why highlighting does not return anything but the document 
id.
I created a core imported all my data, everything seems like it should be 
working.
From reading the documentation I expect it to show me highlight information for
assetName around Potter, but I never get anything but the document id (assetId)

Here are the entries I added to managed-schema for assetName. Anybody know what
I should be targeting as a solution.




#- My query and results -#




  0
  1
  
Potter
on


  clientId:11
  assetTypeId:1


1555116259160
  


  
3
Harry Potter and The Order of The Phoenix.mov
2012-09-27T02:34:27Z
Quicktime with Audio
1
11
Level 3
27
27
Harry Potter and The Order of The Phoenix.mov
23.976
Non drop frame
2.35
16
4:2:0
1624857205677228032
  
3
StudelyCastleHotel
2019-04-12T22:57:33Z
JPEG
1
11
Level 3
10130
10130

  Producer


  Michael Potter


  7

StudelyCastleHotel
1630650901282684928