Data import handler and no status in web-ui

[Thomas Porschberg]

Hi,

I use DIH in solr-cloud mode (implicit route) in solr6.5.1.
When I start the import it works fine and I see the progress in the logfile.
However, when I click the "Refresh Status" button in the web-ui while the 
import is running
I only see "No information available (idle)". 
So I have to look in the logfile the observe when the import was finished.

In the old solr, non-cloud and non-partitioned, there was a hourglass while the 
import was running.

Any idea?

Best regards
Thomas

Re: Adding a Basic Authentication user fails with 404

[Susheel Kumar]

Please chk if it is not due to 6.5 which is fixed in 6.6

http://issues.apache.org/jira/browse/SOLR-10718

On Tue, Jun 6, 2017 at 8:05 PM, David Parker  wrote:

> Hello,
>
> I am running a stand-alone instance of Solr 6.5 (without ZooKeeper).  I am
> attempting to implement Basic Authentication per the documentation, but
> when I try to use the API to add a user, I get a 404 error.  It seems the
> /admin/authentication API entry point isn't there:
>
> $ curl --user solr:SolrRocks http://localhost:8983/solr/
> admin/authentication
> -H 'Content-type:application/json' -d '{"set-user": {"myuser" :
> "mypasswd"}}'
> 
> 
> 
> Error 404 Not Found
> 
> HTTP ERROR 404
> Problem accessing /solr/admin/authentication. Reason:
> Not Found
> 
> 
>
> But according to the documentation, the API entry point is
> admin/authentication, and it states the following:
>
> "This endpoint is not collection-specific, so users are created for the
> entire Solr cluster. If users need to be restricted to a specific
> collection, that can be done with the authorization rules."
>
> The only thing which stands out to me is "users are created for the entire
> Solr cluster."  Is this entry point missing because I'm running Solr
> stand-alone?
>
> Any help is greatly appreciated!
>
> - Dave
>
> --
> Dave Parker
> Database & Systems Administrator
> Utica College
> Integrated Information Technology Services
> (315) 792-3229
> Registered Linux User #408177
>

Re: Why do Solr nodes go into Recovery status

[suresh pendap]

Thanks Erick for the reply.

When the leader asks the follower to go into recovery status,  does it stop
sending future updates to this replica until it becomes fully in sync with
the leader?

Regards
Suresh

On Mon, Jun 5, 2017 at 8:32 PM, Erick Erickson 
wrote:

> bq: This means that technically the replica nodes should not fall behind
> and do
> not have to go into recovery mode
>
> Well, true if nothing weird happens. By "weird" I mean anything that
> interferes with the leader getting anything other than a success code
> back from a follower it sends  document to.
>
> bq: Is this the only scenario in which a node can go into recovery status?
>
> No, there are others. One for-instance: Leader sends a doc to the
> follower and the request times out (huge  GC pauses, the doc takes too
> long to index for whatever reason etc). The leader then sends a
> message to the follower to go directly into the recovery state since
> the leader has no way of knowing whether the follower successfully
> wrote the document to it's transaction log. You'll see messages about
> "leader initiated recovery" in the follower's solr log in this case.
>
> two bits of pedantry:
>
> bq:  Down by the other replicas
>
> Almost. we're talking indexing here and IIUC only the leader can send
> another node into recovery as all updates go through the leader.
>
> If I'm going to be nit-picky, Zookeeper can _also_ cause a node to be
> marked as down if it's periodic ping of the node fails to return.
> Actually I think this is done through another Solr node that ZK
> notifies
>
> bq: It goes into a recovery mode and tries to recover all the
> documents from the leader of shard1.
>
> Also nit-picky. But if the follower isn't "too far" behind it can be
> brought back into sync from via "peer sync" where it gets the missed
> docs sent to it from the tlog of a healthy replica. "Too far" is 100
> docs by default, but can be set in solrconfig.xml if necessary. If
> that limit is exceeded, then indeed the entire index is copied from
> the leader.
>
> Best,
> Erick
>
>
>
> On Mon, Jun 5, 2017 at 5:18 PM, suresh pendap 
> wrote:
> > Hi,
> >
> > Why and in what scenarios do Solr nodes go into recovery status?
> >
> > Given that Solr is a CP system it means that the writes for a Document
> > index are acknowledged only after they are propagated and acknowledged by
> > all the replicas of the Shard.
> >
> > This means that technically the replica nodes should not fall behind and
> do
> > not have to go into recovery mode.
> >
> > Is my above understanding correct?
> >
> > Can a below scenario happen?
> >
> > 1. Assume that we have 3 replicas for Shard shard1 with the names
> > shard1_replica1, shard1_replica2 and shard1_replica3.
> >
> > 2. Due to some reason, network issue or something else, the
> shard1_replica2
> > is not reachable by the other replicas and it is marked as Down by the
> > other replicas (shard1_replica1 and shard1_replica3 in this case)
> >
> > 3. The network issue is restored and the shard1_replica2 is reachable
> > again. It goes into a recovery mode and tries to recover all the
> documents
> > from the leader of shard1.
> >
> > Is this the only scenario in which a node can go into recovery status?
> >
> > In other words, does the node has to go into a Down status before getting
> > back into a recovery status?
> >
> >
> > Regards
>

Re: Why do Solr nodes go into Recovery status

[suresh pendap]

GW,
Did you mean a separate transaction log on Solr or on Zookeeper?

-suresh

On Tue, Jun 6, 2017 at 5:23 AM, GW  wrote:

> I've heard of systems tanking like this on Windows during OS updates.
> Because of this, I run all my updates in attendance even though I'm Linux.
> My Nodes run as VM's, I shut down Solr gracefully, snap shot a backup of
> the VM, update and run. If things go screwy I can always roll back. To me
> it sounds like a lack of resources or a kink in your networking, assuming
> your set up is correct. Watch for home made network cables. I've seen soft
> crimp connectors put on solid wire which can wreck a switch port forever.
> Do you have a separate transaction log device on each Zookeeper? I made
> this mistake in the beginning and had similar problems under load.
>
>
> GW
>
> On 5 June 2017 at 22:32, Erick Erickson  wrote:
>
> > bq: This means that technically the replica nodes should not fall behind
> > and do
> > not have to go into recovery mode
> >
> > Well, true if nothing weird happens. By "weird" I mean anything that
> > interferes with the leader getting anything other than a success code
> > back from a follower it sends  document to.
> >
> > bq: Is this the only scenario in which a node can go into recovery
> status?
> >
> > No, there are others. One for-instance: Leader sends a doc to the
> > follower and the request times out (huge  GC pauses, the doc takes too
> > long to index for whatever reason etc). The leader then sends a
> > message to the follower to go directly into the recovery state since
> > the leader has no way of knowing whether the follower successfully
> > wrote the document to it's transaction log. You'll see messages about
> > "leader initiated recovery" in the follower's solr log in this case.
> >
> > two bits of pedantry:
> >
> > bq:  Down by the other replicas
> >
> > Almost. we're talking indexing here and IIUC only the leader can send
> > another node into recovery as all updates go through the leader.
> >
> > If I'm going to be nit-picky, Zookeeper can _also_ cause a node to be
> > marked as down if it's periodic ping of the node fails to return.
> > Actually I think this is done through another Solr node that ZK
> > notifies
> >
> > bq: It goes into a recovery mode and tries to recover all the
> > documents from the leader of shard1.
> >
> > Also nit-picky. But if the follower isn't "too far" behind it can be
> > brought back into sync from via "peer sync" where it gets the missed
> > docs sent to it from the tlog of a healthy replica. "Too far" is 100
> > docs by default, but can be set in solrconfig.xml if necessary. If
> > that limit is exceeded, then indeed the entire index is copied from
> > the leader.
> >
> > Best,
> > Erick
> >
> >
> >
> > On Mon, Jun 5, 2017 at 5:18 PM, suresh pendap 
> > wrote:
> > > Hi,
> > >
> > > Why and in what scenarios do Solr nodes go into recovery status?
> > >
> > > Given that Solr is a CP system it means that the writes for a Document
> > > index are acknowledged only after they are propagated and acknowledged
> by
> > > all the replicas of the Shard.
> > >
> > > This means that technically the replica nodes should not fall behind
> and
> > do
> > > not have to go into recovery mode.
> > >
> > > Is my above understanding correct?
> > >
> > > Can a below scenario happen?
> > >
> > > 1. Assume that we have 3 replicas for Shard shard1 with the names
> > > shard1_replica1, shard1_replica2 and shard1_replica3.
> > >
> > > 2. Due to some reason, network issue or something else, the
> > shard1_replica2
> > > is not reachable by the other replicas and it is marked as Down by the
> > > other replicas (shard1_replica1 and shard1_replica3 in this case)
> > >
> > > 3. The network issue is restored and the shard1_replica2 is reachable
> > > again. It goes into a recovery mode and tries to recover all the
> > documents
> > > from the leader of shard1.
> > >
> > > Is this the only scenario in which a node can go into recovery status?
> > >
> > > In other words, does the node has to go into a Down status before
> getting
> > > back into a recovery status?
> > >
> > >
> > > Regards
> >
>

Adding a Basic Authentication user fails with 404

[David Parker]

Hello,

I am running a stand-alone instance of Solr 6.5 (without ZooKeeper).  I am
attempting to implement Basic Authentication per the documentation, but
when I try to use the API to add a user, I get a 404 error.  It seems the
/admin/authentication API entry point isn't there:

$ curl --user solr:SolrRocks http://localhost:8983/solr/admin/authentication
-H 'Content-type:application/json' -d '{"set-user": {"myuser" :
"mypasswd"}}'



Error 404 Not Found

HTTP ERROR 404
Problem accessing /solr/admin/authentication. Reason:
Not Found



But according to the documentation, the API entry point is
admin/authentication, and it states the following:

"This endpoint is not collection-specific, so users are created for the
entire Solr cluster. If users need to be restricted to a specific
collection, that can be done with the authorization rules."

The only thing which stands out to me is "users are created for the entire
Solr cluster."  Is this entry point missing because I'm running Solr
stand-alone?

Any help is greatly appreciated!

- Dave

-- 
Dave Parker
Database & Systems Administrator
Utica College
Integrated Information Technology Services
(315) 792-3229
Registered Linux User #408177

RE: Got a 404 trying to update a solr. 6.5.1 server. /solr/update not found.

[Phil Scadden]

Duh! Thanks for that.

-Original Message-
From: tflo...@apple.com [mailto:tflo...@apple.com]
Sent: Tuesday, 6 June 2017 4:25 p.m.
To: solr-user@lucene.apache.org
Subject: Re: Got a 404 trying to update a solr. 6.5.1 server. /solr/update not 
found.

I think you are missing the collection name in the path.

Tomás

Sent from my iPhone

> On Jun 5, 2017, at 9:08 PM, Phil Scadden  wrote:
>
> Simple piece of code. Had been working earlier (though against a 6.4.2 
> instance).
>
>  ConcurrentUpdateSolrClient solr = new 
> ConcurrentUpdateSolrClient("http://myhost:8983/solr",10,2);
>   try {
>solr.deleteByQuery("*:*");
>solr.commit();
>   } catch (SolrServerException | IOException ex) {
>// logger handler stuff omitted.
>   }
>
> Comes back with:
> 15:53:36,693 DEBUG wire:72 -  << "[\n]"
> 15:53:36,694 DEBUG wire:72 -  << " content="text/html;charset=utf-8"/>[\n]"
> 15:53:36,694 DEBUG wire:72 -  << "Error 404 Not Found[\n]"
> 15:53:36,695 DEBUG wire:72 -  << "[\n]"
> 15:53:36,695 DEBUG wire:72 -  << "HTTP ERROR 404[\n]"
> 15:53:36,696 DEBUG wire:72 -  << "Problem accessing /solr/update. 
> Reason:[\n]"
> 15:53:36,696 DEBUG wire:72 -  << "Not Found[\n]"
> 15:53:36,696 DEBUG wire:72 -  << "[\n]"
> 15:53:36,697 DEBUG wire:72 -  << "[\n]"
>
> If I access http://myhost:8983/solr/update then I get that html too, but 
> http://myhost:8983/solr comes up with admin page as normal so Solr appears to 
> be running okay.
> Notice: This email and any attachments are confidential and may not be used, 
> published or redistributed without the prior written consent of the Institute 
> of Geological and Nuclear Sciences Limited (GNS Science). If received in 
> error please destroy and immediately notify GNS Science. Do not copy or 
> disclose the contents.
Notice: This email and any attachments are confidential and may not be used, 
published or redistributed without the prior written consent of the Institute 
of Geological and Nuclear Sciences Limited (GNS Science). If received in error 
please destroy and immediately notify GNS Science. Do not copy or disclose the 
contents.

Re: dynamic fields during segment merge

[Erick Erickson]

Merging has nothing to do with tlogs. Tlogs are simply the last N
(configurable) raw documents to be used for certain index recovery
operations.

Dynamic fields are no different from static fields. In either case the
first time a document is encountered with a field that matches, the
same mechanisms take over so I'm not sure what "cost" you're talking
about here

There are costs associated with having 1,000 fields, but whether
they're dynamic or static fields doesn't matter.

Best,
Erick

On Tue, Jun 6, 2017 at 12:15 PM, Shawn Feldman  wrote:
> When solr is merging segments of the tlog what impact do dynamic fields
> have?  If i have 1k dynamic fields do i pay the cost on every merge or only
> if the documents have those fields?
>
> -shawn

Re: Anonymous Read?

[Solr User]

Thanks!  The null role value did the trick.  I tried this with the
predefined permissions and it worked as well.  Thanks again!

On Tue, Jun 6, 2017 at 2:08 PM, Oakley, Craig (NIH/NLM/NCBI) [C] <
craig.oak...@nih.gov> wrote:

> We usually end security.json with the permissions
>
>{
> "name":"open_select",
>  "path":"/select/*",
>  "role":null},
>  {
> "name":"all-admin",
> "collection":null,
> "path":"/*",
> "role":"allgen"},
>  {
> "name":"all-core-handlers",
> "path":"/*",
>  "role":"allgen"}]
>  } }
>
>
> ...and then assign the "allgen" role to all users
>
> This allows a select without a login & password, but requires a login &
> password for anything else (including the front page of the GUI)
>
> -Original Message-
> From: Solr User [mailto:solr...@gmail.com]
> Sent: Tuesday, June 06, 2017 2:27 PM
> To: solr-user@lucene.apache.org
> Subject: Anonymous Read?
>
> Is it possible to setup Solr security to allow anonymous query (/select
> etc.) but restricted access to other permissions as described in
> https://lucidworks.com/2015/08/17/securing-solr-basic-
> auth-permission-rules/
> ?
>

Re: How to do CDCR with basic auth?

[Shawn Feldman]

looks like this ticket was fixed in 6.6 SOLR-10718


On Fri, May 19, 2017 at 3:19 PM Shawn Feldman 
wrote:

> i added a ticket
>
> https://issues.apache.org/jira/browse/SOLR-10718
>
> we'll see what happens
>
> On Fri, May 19, 2017 at 3:03 PM Shawn Feldman 
> wrote:
>
>> I have the same exact issue on my box.  Basic auth works in 6.4.2 but
>> fails in 6.5.1.  I assume its a bug.  probably just hasn't been
>> acknowledged yet.
>>
>> On Sun, May 14, 2017 at 2:37 PM Xie, Sean  wrote:
>>
>>> Configured the JVM:
>>>
>>> -Dsolr.httpclient.builder.factory=org.apache.solr.client.solrj.impl.PreemptiveBasicAuthConfigurer
>>> -Dbasicauth=solr:SolrRocks
>>>
>>> Configured the CDCR.
>>>
>>> Started the Source cluster and
>>> Getting the log:
>>>
>>> .a.s.h.CdcrUpdateLogSynchronizer Caught unexpected exception
>>> java.lang.IllegalArgumentException: Credentials may not be null
>>> at org.apache.http.util.Args.notNull(Args.java:54)
>>> at org.apache.http.auth.AuthState.update(AuthState.java:113)
>>> at
>>> org.apache.solr.client.solrj.impl.PreemptiveAuth.process(PreemptiveAuth.java:56)
>>> at
>>> org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:132)
>>> at
>>> org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:166)
>>> at
>>> org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:485)
>>> at
>>> org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
>>> at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
>>> at
>>> org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
>>> at
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.java:515)
>>> at
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:279)
>>> at
>>> org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:268)
>>> at
>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
>>> at
>>> org.apache.solr.handler.CdcrUpdateLogSynchronizer$UpdateLogSynchronisation.run(CdcrUpdateLogSynchronizer.java:146)
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> at
>>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
>>> at
>>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> at java.lang.Thread.run(Thread.java:748)
>>>
>>>
>>> Somehow, the cdcr didn’t pickup the credentials when using the
>>> PreemptiveAuth.
>>>
>>> Is it a bug?
>>>
>>> Thanks
>>> Sean
>>>
>>>
>>>
>>> On 5/14/17, 3:09 PM, "Xie, Sean"  wrote:
>>>
>>> So I have configured two clusters (source and target) with basic
>>> auth with solr:SolrRocks, but when starting the source node, log is showing
>>> it couldn’t read the authentication info.
>>>
>>> I already added the –Dbasicauth=solr:SolrRocks to the JVM of the
>>> solr instance. Not sure where else I can configure the solr to use the auth.
>>>
>>> When starting the CDCR, the log is:
>>>
>>> 2017-05-14 15:01:02.915 WARN  (qtp1348949648-21) [c:COL1 s:shard1
>>> r:core_node2 x:COL1_shard1_replica2] o.a.s.h.CdcrReplicatorManager Unable
>>> to instantiate the log reader for target collection COL1
>>> org.apache.solr.client.solrj.SolrServerException:
>>> java.lang.IllegalArgumentException: Credentials may not be null
>>> at
>>> org.apache.solr.client.solrj.impl.LBHttpSolrClient.doRequest(LBHttpSolrClient.java:473)
>>> at
>>> org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:387)
>>> at
>>> org.apache.solr.client.solrj.impl.CloudSolrClient.sendRequest(CloudSolrClient.java:1376)
>>> at
>>> org.apache.solr.client.solrj.impl.CloudSolrClient.requestWithRetryOnStaleState(CloudSolrClient.java:1127)
>>> at
>>> org.apache.solr.client.solrj.impl.CloudSolrClient.request(CloudSolrClient.java:1057)
>>> at
>>> org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219)
>>> at
>>> org.apache.solr.handler.CdcrReplicatorManager.getCheckpoint(CdcrReplicatorManager.java:196)
>>> at
>>> 

dynamic fields during segment merge

[Shawn Feldman]

When solr is merging segments of the tlog what impact do dynamic fields
have?  If i have 1k dynamic fields do i pay the cost on every merge or only
if the documents have those fields?

-shawn

RE: Anonymous Read?

[Oakley, Craig (NIH/NLM/NCBI) [C]]

We usually end security.json with the permissions

   {
"name":"open_select",
 "path":"/select/*",
 "role":null},
 {
"name":"all-admin",
"collection":null,
"path":"/*",
"role":"allgen"},
 {
"name":"all-core-handlers",
"path":"/*",
 "role":"allgen"}]
 } }


...and then assign the "allgen" role to all users

This allows a select without a login & password, but requires a login & 
password for anything else (including the front page of the GUI)

-Original Message-
From: Solr User [mailto:solr...@gmail.com] 
Sent: Tuesday, June 06, 2017 2:27 PM
To: solr-user@lucene.apache.org
Subject: Anonymous Read?

Is it possible to setup Solr security to allow anonymous query (/select
etc.) but restricted access to other permissions as described in
https://lucidworks.com/2015/08/17/securing-solr-basic-auth-permission-rules/
?

Anonymous Read?

[Solr User]

Is it possible to setup Solr security to allow anonymous query (/select
etc.) but restricted access to other permissions as described in
https://lucidworks.com/2015/08/17/securing-solr-basic-auth-permission-rules/
?

Re: Different DateTime format in dataimport and index

[SOLR4189]

I don't use DB. I do dataimport from one collection of SOLR to another
collection with the same configuration. 



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Different-DateTime-format-in-dataimport-and-index-tp4339230p4339244.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr installdir deleted after set up solr cloud

[Erick Erickson]

Ouch! Thanks for letting us know, I can sleep easier now ;)

On Tue, Jun 6, 2017 at 9:12 AM, tstusr  wrote:
> Hi, there
>
> We've got a silly but terrible mistake.
>
> We replace on solrconfig.xml ${solr.data.dir:} for ${solr.install.dir}.
>
> So, when solr cloud config works it replaces on install dir, erasing all
> solr instance.
>
> Thanks for your help.
>
>
>
>
>
>
>
> --
> View this message in context: 
> http://lucene.472066.n3.nabble.com/Solr-installdir-deleted-after-set-up-solr-cloud-tp4338305p4339238.html
> Sent from the Solr - User mailing list archive at Nabble.com.

Re: Solr installdir deleted after set up solr cloud

[tstusr]

Hi, there

We've got a silly but terrible mistake. 

We replace on solrconfig.xml ${solr.data.dir:} for ${solr.install.dir}. 

So, when solr cloud config works it replaces on install dir, erasing all
solr instance.

Thanks for your help.







--
View this message in context: 
http://lucene.472066.n3.nabble.com/Solr-installdir-deleted-after-set-up-solr-cloud-tp4338305p4339238.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Different DateTime format in dataimport and index

[Erick Erickson]

My guess would be that your DIH query is interacting weirdly with your db
and doing time one transforms before Solr gets it.

On Jun 6, 2017 8:38 AM, "SOLR4189"  wrote:

> Let's say I have SolrDoc:
> *{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*,
> where type of id is int, type of price is float, type of name is string and
> type of pickupTime is tdate/date. And let's say I have my update processor
> that writes to log indexed item.
>
> So, my question is why in indexing of item I see in log:
> *{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*
> and in reindex or dataimport I see in log:
> *{id: test1, price: 100.0, name: pizza, pickupTime: Tue Jun 6 19:00:00 IDT
> 2017}*
>
> Why do float and date have different format in index and dataimport? Is it
> SOLR bug?
> How can I change dataimport format to index format?
> Which are types have different format like float and date?
>
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Different-DateTime-format-in-dataimport-and-
> index-tp4339230.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>

Different DateTime format in dataimport and index

[SOLR4189]

Let's say I have SolrDoc: 
*{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*,
where type of id is int, type of price is float, type of name is string and
type of pickupTime is tdate/date. And let's say I have my update processor
that writes to log indexed item. 

So, my question is why in indexing of item I see in log:
*{id: test1, price: 100, name: pizza, pickupTime: 2017-06-06T19:00:00}*
and in reindex or dataimport I see in log:
*{id: test1, price: 100.0, name: pizza, pickupTime: Tue Jun 6 19:00:00 IDT
2017}*

Why do float and date have different format in index and dataimport? Is it
SOLR bug?
How can I change dataimport format to index format?
Which are types have different format like float and date?




--
View this message in context: 
http://lucene.472066.n3.nabble.com/Different-DateTime-format-in-dataimport-and-index-tp4339230.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)

[Varun Thacker]

Does this happen on a fresh Solr 6.3 ( as mentioned on SOLR-10806 ) or was
the index existing with some other version and then upgraded to 6.3 ?

Is the problem reproducible for you?


On Tue, Jun 6, 2017 at 7:26 AM, S G  wrote:

> Hi,
>
> We are seeing some very bad performance on our performance test that tries
> to load a 2 shard, 3 replica system with about 2000 writes/sec and 2000
> reads/sec
>
> The exception stack trace seems to point to a specific line of code and a
> similar stack trace is reported by users on Elastic-Search forums too.
>
> Could this be a a common bug in Lucene which is affecting both the systems?
> https://issues.apache.org/jira/browse/SOLR-10806
>
> One bad part about Solr is that once it happens, the whole system comes to
> a grinding halt.
> Solr UI is not accessible, even for the nodes not hosting any collections !
> It would be really nice to get rid of such an instability in the system.
>
> Thanks
> SG
>
>
>

Re: Facets delayed several seconds after commit

[Erick Erickson]

Most like this is autowarming. New searchers are not available until
the autowarming period is complete. The sequence is:
> commit
> new searcher is opened and autowarming starts on it.
> new requests are served by the old searcher
> autowarming completes
> new requests are served by the new searcher. It's only at this point that any 
> new data from the last commit is visible.
> old searcher finishes last request outstanding on it and is closed.

The Solr logs will give you some insight into how long searchers take
to become active. Also check your autowarm settings, people often make
these far too high.

So my guess is that this is perfectly normal behavior, I suspect you
have some kind of automated test system that queries immediately after
issuing a commit perhaps? You can specify options like waitForSearcher
(IIRC) to NOT return from the commit until the new searcher is active
to avoid this if it's really necessary.

Best,
Erick

On Tue, Jun 6, 2017 at 8:03 AM, Yitzchak Schaffer
 wrote:
> We've observed that, following a commit, it may take several seconds before
> the indexed documents are reflected in facets on subsequent queries. I have
> been able to reproduce this using enum and fc as facet.method
>
> Is this related to the filterCache perhaps? My next step is to test with
> filterCache disabled.
>
> Thank you!
>
> Yitz Schaffer
> Vitals

Facets delayed several seconds after commit

[Yitzchak Schaffer]

We've observed that, following a commit, it may take several seconds before
the indexed documents are reflected in facets on subsequent queries. I have
been able to reproduce this using enum and fc as facet.method

Is this related to the filterCache perhaps? My next step is to test with
filterCache disabled.

Thank you!

Yitz Schaffer
Vitals

Invalid shift value (64) in prefixCoded bytes (is encoded value really an INT?)

[S G]

Hi,

We are seeing some very bad performance on our performance test that tries
to load a 2 shard, 3 replica system with about 2000 writes/sec and 2000
reads/sec

The exception stack trace seems to point to a specific line of code and a
similar stack trace is reported by users on Elastic-Search forums too.

Could this be a a common bug in Lucene which is affecting both the systems?
https://issues.apache.org/jira/browse/SOLR-10806

One bad part about Solr is that once it happens, the whole system comes to
a grinding halt.
Solr UI is not accessible, even for the nodes not hosting any collections !
It would be really nice to get rid of such an instability in the system.

Thanks
SG

Re: Slow inserting with SolrCloud when increasing replicas

[Susheel Kumar]

There is some indexing performance penalty is by adding a replica but then
onwards adding more replica(s) shouldn't have much difference.  So
something very wrong.  Can you watch your logs when indexing happening and
see what's going when gets replicated etc.

Thanks,
Susheel

On Tue, Jun 6, 2017 at 9:28 AM, Isart Montane 
wrote:

> Hi Sisheel!
>
> We are using Solr 6.5.
>
> We've already looked at Issue 7333, but none of the params seem to change
> the behaviour.
>
> Also, I'm not sure having more parallelism will improve performance since
> the problem seems to be related to replication. It looks like the writes
> need to get to all the replicas before the indexing can continue with the
> next batch
>
> Isart
>
>
>
> On Tue, Jun 6, 2017 at 2:31 PM, Susheel Kumar 
> wrote:
>
> > Which version of Solr are you using. See
> >
> > https://lucidworks.com/2015/06/10/indexing-performance-solr-
> > 5-2-now-twice-fast/
> >
> >
> > https://issues.apache.org/jira/browse/SOLR-7333
> >
> > Also would suggest to index using SolrJ with parallelism (multiple
> threads
> > and/or machines) to increase indexing thru-put further.
> >
> > On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane 
> > wrote:
> >
> > > Hello,
> > >
> > > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> > > problem we are seeing is a huge drop on writes when the number of
> > replicas
> > > increase.
> > >
> > > When we index (using DIH and batches) a collection with no replicas, we
> > are
> > > able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> > > replica, 800 with 2 replicas and 400 with 3 replicas and it keeps
> getting
> > > worst when more replicas are added.
> > >
> > > We've been reading about it and it seems that the `replicationFactor`
> > plays
> > > a big role on that, but we've got it set to 1, so I'm not sure why it
> > keeps
> > > decreasing when more replicas are added. In fact, we don't need the
> data
> > to
> > > be replicated in real time (we can even afford minutes of delay), but
> > I've
> > > been unable to find how to tune that.
> > >
> > > Has anyone  experienced a similar behaviour? is there any way to
> increase
> > > the indexing performance when using SolrCloud?
> > >
> > > We've seen posts about people having +100 replicas, so my feeling is
> that
> > > there's something to tune that we are not doing.
> > >
> > > Thanks
> > >
> > >
> > > Isart Montane Mogas
> > >
> >
>

Re: Slow inserting with SolrCloud when increasing replicas

[Isart Montane]

Hi Sisheel!

We are using Solr 6.5.

We've already looked at Issue 7333, but none of the params seem to change
the behaviour.

Also, I'm not sure having more parallelism will improve performance since
the problem seems to be related to replication. It looks like the writes
need to get to all the replicas before the indexing can continue with the
next batch

Isart



On Tue, Jun 6, 2017 at 2:31 PM, Susheel Kumar  wrote:

> Which version of Solr are you using. See
>
> https://lucidworks.com/2015/06/10/indexing-performance-solr-
> 5-2-now-twice-fast/
>
>
> https://issues.apache.org/jira/browse/SOLR-7333
>
> Also would suggest to index using SolrJ with parallelism (multiple threads
> and/or machines) to increase indexing thru-put further.
>
> On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane 
> wrote:
>
> > Hello,
> >
> > We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> > problem we are seeing is a huge drop on writes when the number of
> replicas
> > increase.
> >
> > When we index (using DIH and batches) a collection with no replicas, we
> are
> > able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> > replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
> > worst when more replicas are added.
> >
> > We've been reading about it and it seems that the `replicationFactor`
> plays
> > a big role on that, but we've got it set to 1, so I'm not sure why it
> keeps
> > decreasing when more replicas are added. In fact, we don't need the data
> to
> > be replicated in real time (we can even afford minutes of delay), but
> I've
> > been unable to find how to tune that.
> >
> > Has anyone  experienced a similar behaviour? is there any way to increase
> > the indexing performance when using SolrCloud?
> >
> > We've seen posts about people having +100 replicas, so my feeling is that
> > there's something to tune that we are not doing.
> >
> > Thanks
> >
> >
> > Isart Montane Mogas
> >
>

Re: Slow inserting with SolrCloud when increasing replicas

[Susheel Kumar]

Which version of Solr are you using. See

https://lucidworks.com/2015/06/10/indexing-performance-solr-5-2-now-twice-fast/


https://issues.apache.org/jira/browse/SOLR-7333

Also would suggest to index using SolrJ with parallelism (multiple threads
and/or machines) to increase indexing thru-put further.

On Tue, Jun 6, 2017 at 4:51 AM, Isart Montane 
wrote:

> Hello,
>
> We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
> problem we are seeing is a huge drop on writes when the number of replicas
> increase.
>
> When we index (using DIH and batches) a collection with no replicas, we are
> able to index at 1800 inserts/sec. That number decreases to 1200 with 1
> replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
> worst when more replicas are added.
>
> We've been reading about it and it seems that the `replicationFactor` plays
> a big role on that, but we've got it set to 1, so I'm not sure why it keeps
> decreasing when more replicas are added. In fact, we don't need the data to
> be replicated in real time (we can even afford minutes of delay), but I've
> been unable to find how to tune that.
>
> Has anyone  experienced a similar behaviour? is there any way to increase
> the indexing performance when using SolrCloud?
>
> We've seen posts about people having +100 replicas, so my feeling is that
> there's something to tune that we are not doing.
>
> Thanks
>
>
> Isart Montane Mogas
>

Re: Why do Solr nodes go into Recovery status

[GW]

I've heard of systems tanking like this on Windows during OS updates.
Because of this, I run all my updates in attendance even though I'm Linux.
My Nodes run as VM's, I shut down Solr gracefully, snap shot a backup of
the VM, update and run. If things go screwy I can always roll back. To me
it sounds like a lack of resources or a kink in your networking, assuming
your set up is correct. Watch for home made network cables. I've seen soft
crimp connectors put on solid wire which can wreck a switch port forever.
Do you have a separate transaction log device on each Zookeeper? I made
this mistake in the beginning and had similar problems under load.


GW

On 5 June 2017 at 22:32, Erick Erickson  wrote:

> bq: This means that technically the replica nodes should not fall behind
> and do
> not have to go into recovery mode
>
> Well, true if nothing weird happens. By "weird" I mean anything that
> interferes with the leader getting anything other than a success code
> back from a follower it sends  document to.
>
> bq: Is this the only scenario in which a node can go into recovery status?
>
> No, there are others. One for-instance: Leader sends a doc to the
> follower and the request times out (huge  GC pauses, the doc takes too
> long to index for whatever reason etc). The leader then sends a
> message to the follower to go directly into the recovery state since
> the leader has no way of knowing whether the follower successfully
> wrote the document to it's transaction log. You'll see messages about
> "leader initiated recovery" in the follower's solr log in this case.
>
> two bits of pedantry:
>
> bq:  Down by the other replicas
>
> Almost. we're talking indexing here and IIUC only the leader can send
> another node into recovery as all updates go through the leader.
>
> If I'm going to be nit-picky, Zookeeper can _also_ cause a node to be
> marked as down if it's periodic ping of the node fails to return.
> Actually I think this is done through another Solr node that ZK
> notifies
>
> bq: It goes into a recovery mode and tries to recover all the
> documents from the leader of shard1.
>
> Also nit-picky. But if the follower isn't "too far" behind it can be
> brought back into sync from via "peer sync" where it gets the missed
> docs sent to it from the tlog of a healthy replica. "Too far" is 100
> docs by default, but can be set in solrconfig.xml if necessary. If
> that limit is exceeded, then indeed the entire index is copied from
> the leader.
>
> Best,
> Erick
>
>
>
> On Mon, Jun 5, 2017 at 5:18 PM, suresh pendap 
> wrote:
> > Hi,
> >
> > Why and in what scenarios do Solr nodes go into recovery status?
> >
> > Given that Solr is a CP system it means that the writes for a Document
> > index are acknowledged only after they are propagated and acknowledged by
> > all the replicas of the Shard.
> >
> > This means that technically the replica nodes should not fall behind and
> do
> > not have to go into recovery mode.
> >
> > Is my above understanding correct?
> >
> > Can a below scenario happen?
> >
> > 1. Assume that we have 3 replicas for Shard shard1 with the names
> > shard1_replica1, shard1_replica2 and shard1_replica3.
> >
> > 2. Due to some reason, network issue or something else, the
> shard1_replica2
> > is not reachable by the other replicas and it is marked as Down by the
> > other replicas (shard1_replica1 and shard1_replica3 in this case)
> >
> > 3. The network issue is restored and the shard1_replica2 is reachable
> > again. It goes into a recovery mode and tries to recover all the
> documents
> > from the leader of shard1.
> >
> > Is this the only scenario in which a node can go into recovery status?
> >
> > In other words, does the node has to go into a Down status before getting
> > back into a recovery status?
> >
> >
> > Regards
>

Re: I want "john smi" to find "john smith" in my custom "fullname_s" field

[Amrit Sarkar]

Erik,

Thank you for correcting. Things I miss out on daily bases: _text_ :)

Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2

On Tue, Jun 6, 2017 at 5:12 PM, Nick Way 
wrote:

> Fantastic thank you so much; I now have 'fullname_s:#string.
> spacesescaped#*
> or email_s:#string.spacesescaped#*' which is working like a dream - thank
> you so much - really appreciate your help.
>
> Thank you also Amrit.
>
> Nick
>
> On 6 June 2017 at 10:40, Erik Hatcher  wrote:
>
> > Nick - try escaping the space, so that your query is q=fullname_s:john\
> > smi*
> >
> > However, whitespace and escaping is problematic.  There is a handy prefix
> > query parser, so this would work on a string field with spaces:
> >
> > q={!prefix f=fullname_s}john smi
> >
> > note no trailing asterisk on that one.   Even better, IMO, is to separate
> > the query string from the query parser:
> >
> > q={!prefix f=fullname_s v=$qq}=john smi
> >
> > Erik
> >
> > 
> >
> > Amrit - the issue with your example below is that q=fullname_s:john smi*
> > parses “john” against fullname_s and “smi” as a prefix query against the
> > default field, not likely fullname_s.   Check your parsed query to see
> > exactly how it parsed.It works for you because… magic!   (copyField *
> > => _text_)
> >
> >
> >
> >
> > > On Jun 6, 2017, at 5:14 AM, Amrit Sarkar 
> wrote:
> > >
> > > Nick,
> > >
> > > "string" is a primitive data-type and the entire value of a field is
> > > indexed as single token. The regex matching happens against the tokens
> > for
> > > text fields and against the full content for string fields. So once a
> > piece
> > > of text is tokenized, there is no way to perform a regex query across
> > word
> > > boundaries.
> > >
> > > fullname_s:john smi* is working for me.
> > >
> > > {
> > >  "responseHeader":{
> > >"zkConnected":true,
> > >"status":0,
> > >"QTime":16,
> > >"params":{
> > >  "q":"fullname_s:john smi*",
> > >  "indent":"on",
> > >  "wt":"json"}},
> > >  "response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
> > >  {
> > >"id":"1",
> > >"fullname_s":"john smith",
> > >"_version_":1569446064473243648}]
> > >  }}
> > >
> > > I am on Solr 6.5.0. What version you are on?
> > >
> > >
> > > Amrit Sarkar
> > > Search Engineer
> > > Lucidworks, Inc.
> > > 415-589-9269
> > > www.lucidworks.com
> > > Twitter http://twitter.com/lucidworks
> > > LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> > >
> > > On Tue, Jun 6, 2017 at 1:30 PM, Nick Way  >
> > > wrote:
> > >
> > >> Hi - I have a Solr collection with a custom field "fullname_s" (a
> > string).
> > >>
> > >> I want "john smi" to find "john smith" (I lower-cased the names upon
> > >> indexing them)
> > >>
> > >> I have tried
> > >>
> > >> fullname_s:"john smi*"
> > >> fullname_s:john smi*
> > >> fullname_s:"john smi?"
> > >> fullname_s:john smi?
> > >>
> > >>
> > >> but nothing gives the expected result - am I missing something? I
> spent
> > >> hours on this one point yesterday so if anyone can please point me in
> > the
> > >> right direction I'd be really grateful.
> > >>
> > >> I'm using Solr with Adobe Coldfusion by the way but I think the
> > principles
> > >> are the same.
> > >>
> > >> Thank you!
> > >>
> > >> Nick
> > >>
> >
> >
>

Re: I want "john smi" to find "john smith" in my custom "fullname_s" field

[Nick Way]

Fantastic thank you so much; I now have 'fullname_s:#string.spacesescaped#*
or email_s:#string.spacesescaped#*' which is working like a dream - thank
you so much - really appreciate your help.

Thank you also Amrit.

Nick

On 6 June 2017 at 10:40, Erik Hatcher  wrote:

> Nick - try escaping the space, so that your query is q=fullname_s:john\
> smi*
>
> However, whitespace and escaping is problematic.  There is a handy prefix
> query parser, so this would work on a string field with spaces:
>
> q={!prefix f=fullname_s}john smi
>
> note no trailing asterisk on that one.   Even better, IMO, is to separate
> the query string from the query parser:
>
> q={!prefix f=fullname_s v=$qq}=john smi
>
> Erik
>
> 
>
> Amrit - the issue with your example below is that q=fullname_s:john smi*
> parses “john” against fullname_s and “smi” as a prefix query against the
> default field, not likely fullname_s.   Check your parsed query to see
> exactly how it parsed.It works for you because… magic!   (copyField *
> => _text_)
>
>
>
>
> > On Jun 6, 2017, at 5:14 AM, Amrit Sarkar  wrote:
> >
> > Nick,
> >
> > "string" is a primitive data-type and the entire value of a field is
> > indexed as single token. The regex matching happens against the tokens
> for
> > text fields and against the full content for string fields. So once a
> piece
> > of text is tokenized, there is no way to perform a regex query across
> word
> > boundaries.
> >
> > fullname_s:john smi* is working for me.
> >
> > {
> >  "responseHeader":{
> >"zkConnected":true,
> >"status":0,
> >"QTime":16,
> >"params":{
> >  "q":"fullname_s:john smi*",
> >  "indent":"on",
> >  "wt":"json"}},
> >  "response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
> >  {
> >"id":"1",
> >"fullname_s":"john smith",
> >"_version_":1569446064473243648}]
> >  }}
> >
> > I am on Solr 6.5.0. What version you are on?
> >
> >
> > Amrit Sarkar
> > Search Engineer
> > Lucidworks, Inc.
> > 415-589-9269
> > www.lucidworks.com
> > Twitter http://twitter.com/lucidworks
> > LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> >
> > On Tue, Jun 6, 2017 at 1:30 PM, Nick Way 
> > wrote:
> >
> >> Hi - I have a Solr collection with a custom field "fullname_s" (a
> string).
> >>
> >> I want "john smi" to find "john smith" (I lower-cased the names upon
> >> indexing them)
> >>
> >> I have tried
> >>
> >> fullname_s:"john smi*"
> >> fullname_s:john smi*
> >> fullname_s:"john smi?"
> >> fullname_s:john smi?
> >>
> >>
> >> but nothing gives the expected result - am I missing something? I spent
> >> hours on this one point yesterday so if anyone can please point me in
> the
> >> right direction I'd be really grateful.
> >>
> >> I'm using Solr with Adobe Coldfusion by the way but I think the
> principles
> >> are the same.
> >>
> >> Thank you!
> >>
> >> Nick
> >>
>
>

Re: Velocity UI with Analyzing Infix Suggester?

[Rick Leir]

> typeahead solutions using a separate collection

Erik, Do you use a separate collection so it can be smaller and thereby faster? 
Or so you can keep good performance on the the main collection server? In my 
mind, the performance of the as-you-type is more important than the regular 
search.
Cheers -- Rick

On June 6, 2017 5:31:08 AM EDT, Erik Hatcher  wrote:
>Walter -
>
>I’ve done several one-off demos that have incorporated as-you-type Ajax
>actions into /browse.   The first one I did was “instant search” (not
>suggest) and left that sitting over at my “instant_search” branch - of
>svn(!).  See the top two commits listed here:
>https://github.com/erikhatcher/lucene-solr-svn/commits/instant_search
>
>Lately I’ve been building typeahead solutions using a separate
>collection rather than the Suggester component and wiring that into
>/browse with just this sort of thing:
>
>$(function() { $(‘#search_box').bind("keyup",load_results); });
>
>where load_results() does this:
>
>  $(‘#results’).load(…url with q=…)
>
>It’s awesome to hear you use wt=velocity - made my day!   And by “in
>6.5.1” you mean it is in the way old tech products configset where it
>uses an ancient jquery.autocomplete feature.  You could probably adapt
>that bit straightforwardly to another endpoint and adjusting the
>`extraParams` in there appropriately.  The trick used here is that the
>response from /terms is simply a single suggestion per line in plain
>text, by way of using wt=velocity with v.template=suggest:
>
>#foreach($t in $response.response.terms.name)
>  $t.key
>#end
>
>Adjust that template to deal with your suggester end-point response so
>that it writes out one per line as plain text and you’re there.   
>Happy to help further if you run into any issues.
>
>And yes, it’d be nice if this got built-in more modernly into the out
>of the box /browse.  If you want to open a JIRA and hack through it
>together I’m game.
>
>   Erik
>
>
>> On Jun 5, 2017, at 4:14 PM, Walter Underwood 
>wrote:
>> 
>> Does anyone have the new suggester working in the Velocity browse UI?
>In 6.5.1, it uses the terms component.
>> 
>> I could probably figure out how to do that in Velocity, but if
>someone has already done that, it would be great.
>> 
>> We use the Velocity UI as an internal exploration and diagnostic
>search page.
>> 
>> wunder
>> Walter Underwood
>> wun...@wunderwood.org
>> http://observer.wunderwood.org/  (my blog)
>> 
>> 

-- 
Sorry for being brief. Alternate email is rickleir at yahoo dot com 

Re: I want "john smi" to find "john smith" in my custom "fullname_s" field

[Erik Hatcher]

Nick - try escaping the space, so that your query is q=fullname_s:john\ smi* 

However, whitespace and escaping is problematic.  There is a handy prefix query 
parser, so this would work on a string field with spaces:

q={!prefix f=fullname_s}john smi

note no trailing asterisk on that one.   Even better, IMO, is to separate the 
query string from the query parser:

q={!prefix f=fullname_s v=$qq}=john smi

Erik



Amrit - the issue with your example below is that q=fullname_s:john smi* parses 
“john” against fullname_s and “smi” as a prefix query against the default 
field, not likely fullname_s.   Check your parsed query to see exactly how it 
parsed.It works for you because… magic!   (copyField * => _text_)




> On Jun 6, 2017, at 5:14 AM, Amrit Sarkar  wrote:
> 
> Nick,
> 
> "string" is a primitive data-type and the entire value of a field is
> indexed as single token. The regex matching happens against the tokens for
> text fields and against the full content for string fields. So once a piece
> of text is tokenized, there is no way to perform a regex query across word
> boundaries.
> 
> fullname_s:john smi* is working for me.
> 
> {
>  "responseHeader":{
>"zkConnected":true,
>"status":0,
>"QTime":16,
>"params":{
>  "q":"fullname_s:john smi*",
>  "indent":"on",
>  "wt":"json"}},
>  "response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
>  {
>"id":"1",
>"fullname_s":"john smith",
>"_version_":1569446064473243648}]
>  }}
> 
> I am on Solr 6.5.0. What version you are on?
> 
> 
> Amrit Sarkar
> Search Engineer
> Lucidworks, Inc.
> 415-589-9269
> www.lucidworks.com
> Twitter http://twitter.com/lucidworks
> LinkedIn: https://www.linkedin.com/in/sarkaramrit2
> 
> On Tue, Jun 6, 2017 at 1:30 PM, Nick Way 
> wrote:
> 
>> Hi - I have a Solr collection with a custom field "fullname_s" (a string).
>> 
>> I want "john smi" to find "john smith" (I lower-cased the names upon
>> indexing them)
>> 
>> I have tried
>> 
>> fullname_s:"john smi*"
>> fullname_s:john smi*
>> fullname_s:"john smi?"
>> fullname_s:john smi?
>> 
>> 
>> but nothing gives the expected result - am I missing something? I spent
>> hours on this one point yesterday so if anyone can please point me in the
>> right direction I'd be really grateful.
>> 
>> I'm using Solr with Adobe Coldfusion by the way but I think the principles
>> are the same.
>> 
>> Thank you!
>> 
>> Nick
>> 

Re: Velocity UI with Analyzing Infix Suggester?

[Erik Hatcher]

Walter -

I’ve done several one-off demos that have incorporated as-you-type Ajax actions 
into /browse.   The first one I did was “instant search” (not suggest) and left 
that sitting over at my “instant_search” branch - of svn(!).  See the top two 
commits listed here: 
https://github.com/erikhatcher/lucene-solr-svn/commits/instant_search

Lately I’ve been building typeahead solutions using a separate collection 
rather than the Suggester component and wiring that into /browse with just this 
sort of thing:

$(function() { $(‘#search_box').bind("keyup",load_results); });

where load_results() does this:

  $(‘#results’).load(…url with q=…)

It’s awesome to hear you use wt=velocity - made my day!   And by “in 6.5.1” you 
mean it is in the way old tech products configset where it uses an ancient 
jquery.autocomplete feature.  You could probably adapt that bit 
straightforwardly to another endpoint and adjusting the `extraParams` in there 
appropriately.  The trick used here is that the response from /terms is simply 
a single suggestion per line in plain text, by way of using wt=velocity with 
v.template=suggest:

#foreach($t in $response.response.terms.name)
  $t.key
#end

Adjust that template to deal with your suggester end-point response so that it 
writes out one per line as plain text and you’re there.Happy to help 
further if you run into any issues.

And yes, it’d be nice if this got built-in more modernly into the out of the 
box /browse.  If you want to open a JIRA and hack through it together I’m game.

Erik


> On Jun 5, 2017, at 4:14 PM, Walter Underwood  wrote:
> 
> Does anyone have the new suggester working in the Velocity browse UI? In 
> 6.5.1, it uses the terms component.
> 
> I could probably figure out how to do that in Velocity, but if someone has 
> already done that, it would be great.
> 
> We use the Velocity UI as an internal exploration and diagnostic search page.
> 
> wunder
> Walter Underwood
> wun...@wunderwood.org
> http://observer.wunderwood.org/  (my blog)
> 
> 

Re: Number of requests spike up, when i do the delta Import.

[vrindavda]

I found this article helpful.

https://wiki.apache.org/solr/DataImportHandlerDeltaQueryViaFullImport



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Number-of-requests-spike-up-when-i-do-the-delta-Import-tp4338162p4339168.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: I want "john smi" to find "john smith" in my custom "fullname_s" field

[Amrit Sarkar]

Nick,

"string" is a primitive data-type and the entire value of a field is
indexed as single token. The regex matching happens against the tokens for
text fields and against the full content for string fields. So once a piece
of text is tokenized, there is no way to perform a regex query across word
boundaries.

fullname_s:john smi* is working for me.

{
  "responseHeader":{
"zkConnected":true,
"status":0,
"QTime":16,
"params":{
  "q":"fullname_s:john smi*",
  "indent":"on",
  "wt":"json"}},
  "response":{"numFound":1,"start":0,"maxScore":1.0,"docs":[
  {
"id":"1",
"fullname_s":"john smith",
"_version_":1569446064473243648}]
  }}

I am on Solr 6.5.0. What version you are on?


Amrit Sarkar
Search Engineer
Lucidworks, Inc.
415-589-9269
www.lucidworks.com
Twitter http://twitter.com/lucidworks
LinkedIn: https://www.linkedin.com/in/sarkaramrit2

On Tue, Jun 6, 2017 at 1:30 PM, Nick Way 
wrote:

> Hi - I have a Solr collection with a custom field "fullname_s" (a string).
>
> I want "john smi" to find "john smith" (I lower-cased the names upon
> indexing them)
>
> I have tried
>
> fullname_s:"john smi*"
> fullname_s:john smi*
> fullname_s:"john smi?"
> fullname_s:john smi?
>
>
> but nothing gives the expected result - am I missing something? I spent
> hours on this one point yesterday so if anyone can please point me in the
> right direction I'd be really grateful.
>
> I'm using Solr with Adobe Coldfusion by the way but I think the principles
> are the same.
>
> Thank you!
>
> Nick
>

Slow inserting with SolrCloud when increasing replicas

[Isart Montane]

Hello,

We are using SolrCloud with 5 nodes, 2 collections, 2 shards each. The
problem we are seeing is a huge drop on writes when the number of replicas
increase.

When we index (using DIH and batches) a collection with no replicas, we are
able to index at 1800 inserts/sec. That number decreases to 1200 with 1
replica, 800 with 2 replicas and 400 with 3 replicas and it keeps getting
worst when more replicas are added.

We've been reading about it and it seems that the `replicationFactor` plays
a big role on that, but we've got it set to 1, so I'm not sure why it keeps
decreasing when more replicas are added. In fact, we don't need the data to
be replicated in real time (we can even afford minutes of delay), but I've
been unable to find how to tune that.

Has anyone  experienced a similar behaviour? is there any way to increase
the indexing performance when using SolrCloud?

We've seen posts about people having +100 replicas, so my feeling is that
there's something to tune that we are not doing.

Thanks


Isart Montane Mogas

I want "john smi" to find "john smith" in my custom "fullname_s" field

[Nick Way]

Hi - I have a Solr collection with a custom field "fullname_s" (a string).

I want "john smi" to find "john smith" (I lower-cased the names upon
indexing them)

I have tried

fullname_s:"john smi*"
fullname_s:john smi*
fullname_s:"john smi?"
fullname_s:john smi?


but nothing gives the expected result - am I missing something? I spent
hours on this one point yesterday so if anyone can please point me in the
right direction I'd be really grateful.

I'm using Solr with Adobe Coldfusion by the way but I think the principles
are the same.

Thank you!

Nick