Re: Does DIH queues up requests

[William Bell]

However you can create multiple DIH configs under a core/collection. You
can run them each in parallel and commit at the end.

SELECT *
 FROM existingtable
 WHERE column >= 1 AND column <= 2000;
SELECT *
 FROM existingtable
 WHERE column >= 2001 AND column <= 4000;


Something like that works for us to speed it up.

On Wed, Jan 25, 2017 at 4:01 PM, Davis, Daniel (NIH/NLM) [C] <
daniel.da...@nih.gov> wrote:

> DIH is not multi-threaded, and so the idea of "queueing" up requests is a
> misnomer.   You might be better off using something other than
> DataImportHandler.
> LogStash can pull what it calls "events" from a database and then push
> them into Solr, and you have some of the same row transformation
> capabilities that DataImportHandler has.
>
> This is also the bread and butter of ETL tools such as
> Kettle/Talend/MuleSoft/etc.
>
> That said, what I have done in the past is to take different streams of
> data and divide them into different requestHandlers, all using
> DataImportHandler.
> Each of these request handlers has its own context as to whether it is
> busy or not, and so each can be separately active/inactive.
>
>   
>class="solr.DataImportHandler">
> 
>   health-topics-conf.xml
> 
>   
>
>   
>   
> 
>   drugs-conf.xml
> 
>   
>
>
> Both of the above or XML imports, but with database imports, I also
> one-time implemented a sort of multithreading by having 4 request handlers
> and 4 data-config files, each taking their own slice of data:
>
> data-config-0.xml
> ...
>  query="SELECT * FROM (SELECT t.*, Mod(RowNum, 4) threadid FROM
> my_data_view t) WHERE threadid = 0"
> transformer="TemplateTransformer,LogTransformer"
> logTemplate="topic thread 0">
> ...
>
> data-config-1.xml:
> ...
>  query="SELECT * FROM (SELECT t.*, Mod(RowNum, 4) threadid FROM
> my_data_view t) WHERE threadid = 1"
> transformer="TemplateTransformer,LogTransformer"
> logTemplate="topic thread 1" logLevel="debug">
> ...
>
> And so on...
>
> -Original Message-
> From: William Bell [mailto:billnb...@gmail.com]
> Sent: Wednesday, January 25, 2017 5:39 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Does DIH queues up requests
>
> What we do is :
>
> Run URL to delete *:*, but do not commit.
>
> 1. Kick off indexing on DIH1, clean=false, commit=false.
> 2. Kick off indexing on DIH2, clean=false, commit=false
>
> Then we manually commit.
>
> On Wed, Jan 25, 2017 at 2:57 PM, Nkeet Shah <nkeet.s...@mathworks.com>
> wrote:
>
> > Hi,
> > I have a multi-thread application that makes DIH request to perform
> > indexing. What I could not gather from the documentation is that does
> > DIH requests are queued up.
> >
> > In essence if a made a request to say DIH1 and it has accepted the
> > request and is working on the indexing. What would happen if another
> > request is made to the same DIH1. Will it be queued or rejected/
> >
> > Thanks
> > Ankit!
> >
> >
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: [Benchmark SOLR] JETTY VS TOMCAT

[William Bell]

Did you try:

Set your acceptor count, SelectChannelConnector.setAcceptors(int)

to
be a a value between 1 and (number_of_cpu_cores - 1).

On Fri, Jan 27, 2017 at 3:22 AM, Gerald Reinhart  wrote:

> Hello,
>
>We are migrating our platform
>from
> - Solr 5.4.1 hosted by a Tomcat
>to
> - Solr 5.4.1 standalone (hosted by Jetty)
>
> => Jetty is 15% slower than Tomcat in the same conditions.
>
>
>   Here are details about the benchmarks :
>
>   Context :
>- Index with 9 000 000 documents
>- Gatling launch queries extracted from the real traffic
>- Server :  R410 with 16 virtual CPU and 96G mem
>
>   Results with 20 clients in // during 10 minutes:
>For Tomcat :
>- 165 Queries per seconds
>- 120ms mean response time
>
>For Jetty :
>- 139 Queries per seconds
>- 142ms mean response time
>
> We have checked :
>   - the load of the server => same
>   - the io wait => same
>   - the memory used in the JVM => same
>   - JVM GC settings => same
>
>  For us, it's a blocker for the migration.
>
>  Is it a known issue ? (I found that :
> http://www.asjava.com/jetty/jetty-vs-tomcat-performance-comparison/)
>
>  How can we improve the performance of Jetty ? (We have already
> followed
> http://www.eclipse.org/jetty/documentation/9.2.21.v20170120/
> optimizing.html
> recommendation)
>
> Many thanks,
>
>
> Gérald Reinhart
>
>
> Kelkoo SAS
> Société par Actions Simplifiée
> Au capital de € 4.168.964,30
> Siège social : 158 Ter Rue du Temple 75003 Paris
> 425 093 069 RCS Paris
>
> Ce message et les pièces jointes sont confidentiels et établis à
> l'attention exclusive de leurs destinataires. Si vous n'êtes pas le
> destinataire de ce message, merci de le détruire et d'en avertir
> l'expéditeur.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Does DIH queues up requests

[William Bell]

What we do is :

Run URL to delete *:*, but do not commit.

1. Kick off indexing on DIH1, clean=false, commit=false.
2. Kick off indexing on DIH2, clean=false, commit=false

Then we manually commit.

On Wed, Jan 25, 2017 at 2:57 PM, Nkeet Shah 
wrote:

> Hi,
> I have a multi-thread application that makes DIH request to perform
> indexing. What I could not gather from the documentation is that does DIH
> requests are queued up.
>
> In essence if a made a request to say DIH1 and it has accepted the request
> and is working on the indexing. What would happen if another request is
> made to the same DIH1. Will it be queued or rejected/
>
> Thanks
> Ankit!
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

How to solve?

[William Bell]

We are entering entries into SOLR like the following, and we want to see if
my pt matches any of these radiuses.

1. Red, pt=39,-107, radius=10km
2. Blue, pt=39,-108, radius=50km

I want to run a SOLR select with a pt=39,-104 and see if it is within 10km
of point 1, and 50km of point 2?

Usually I know you can :

http://localhost:8983/select?q=*:*=39,-104=solr_geohash= ??

One idea was to use bbox and find the N,S,E,W pt for point 1 and point 2.
But this is not idea, we want to use Great Circle.

Thoughts?


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Limit = 0? Does it still calculate facet ?

[William Bell]

Yeah we have a bunch of facet.fields that we need, but want to selectively
turn a few off based on user input.

QT - has default facet.fields.

We want to turn a couple off like this:

http://localhost:8983/solr/core/select?qt=provider=0

Will this turn off the payor field that is defined in QT?

Any other way to do it?

On Thu, Dec 22, 2016 at 4:15 PM, Tomás Fernández Löbbe <
tomasflo...@gmail.com> wrote:

> Yes, facet.limit will short circuit and not calculate the facet for the
> field. I'm assuming you can't just use facet=false?
>
> Tomas
>
> On Thu, Dec 22, 2016 at 1:00 PM, William Bell <billnb...@gmail.com> wrote:
>
> > We have a qt=provider and it sets facets.
> >
> > We want to short circuit the facet. Can we set limit=0 and will it NOT
> > calculate it?
> >
> > Or does it calculate it and not return results? Can we make it faster ?
> >
> > f..facet.limit = 0
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Limit = 0? Does it still calculate facet ?

[William Bell]

We have a qt=provider and it sets facets.

We want to short circuit the facet. Can we set limit=0 and will it NOT
calculate it?

Or does it calculate it and not return results? Can we make it faster ?

f..facet.limit = 0

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: XFS or EXT4 on Amazon AWS AMIs

[William Bell]

http://edgystuff.tumblr.com/post/81219256714/tips-to-check-and-improve-your-storage-io

Which specifies:  SERVER-13417
<http://t.umblr.com/redirect?z=https%3A%2F%2Fjira.mongodb.org%2Fbrowse%2FSERVER-13417=MDk2ZWYxNDBjZjcxNDkwODQ1YmYyNDZjZjU1ZDU0YTlhODE1NmM0MyxXcEY4cXhmTQ%3D%3D=t%3Ax6OZEfmA-wU2yrtBgvccww=http%3A%2F%2Fedgystuff.tumblr.com%2Fpost%2F81219256714%2Ftips-to-check-and-improve-your-storage-io=1>


You might be right on XFS... We are testing today.

On Thu, Dec 22, 2016 at 1:03 AM, Will Martin <wmartin...@outlook.com> wrote:

> I'd like to see the MongoDB report(?). ext4fs design specifications
> includes support for large files via allocation placement. MongoDB, the
> last time I checked, does pre-allocation which gives it the performance
> benefit of ext4fs multiple design factors (Block and Inode Allocation
> Policy), but the disadvantage of having to rebuild when file lengths are
> being exceeded; at which time the disk fragmentation may prevent ext4fs
> from getting the allocation pattern it was designed for.
>
> That design feature is going to be unavailable with Solr where ext4fs
> dynamic allocation features are less deterministic. Other performance
> factors on ext4fs, and mutexes (even with guard mutexes) are pretty
> standard patterns. The threaded calls sound like the advantages of the
> allocation pattern.
>
> Still those statements, *based on a dated reading of mine*, may be out of
> date with the MongoDB report factors.
>
> "ext4 recognizes (better than ext3, anyway) that data locality is
> generally a desirably quality of a filesystem"
>
> https://ext4.wiki.kernel.org/index.php/Ext4_Disk_Layout#
> Block_and_Inode_Allocation_Policy
>
> For AWS AMI, is there an r4 instance type? The c3 and m3 are superseded
> with *4 types that have notable improvements in IOPs and don't cost more.
>
> http://howto.unixdev.net/Test_LVM_Trim_Ext4.html   -- not an extended
> performance benchmark, but useful to validate discard/TRIM.
>
> On 12/22/2016 1:32 AM, William Bell wrote:
>
> So what are people recommending for SOLR on AWS on Amazon AMI - ext4 or
> xfs?
>
> I saw an article about MongoDB - saying performance on Amazon was better
> due to a mutex issue on ext4 files and threaded calls.
>
> I have been using ext4 for a long time, but I am moving to r3.* instances
> and TRIM / DISCARD support just appears more supported on XFS.
>
>
>
>
>
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

XFS or EXT4 on Amazon AWS AMIs

[William Bell]

So what are people recommending for SOLR on AWS on Amazon AMI - ext4 or xfs?

I saw an article about MongoDB - saying performance on Amazon was better
due to a mutex issue on ext4 files and threaded calls.

I have been using ext4 for a long time, but I am moving to r3.* instances
and TRIM / DISCARD support just appears more supported on XFS.




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Caching multiple entities

[William Bell]

I am not sure, but it looks like your XML is invalid.

last_modified > XYZ

You need to switch to   or use something like a database view so that
the > and other < will not cause problems.


On Sat, Dec 17, 2016 at 7:01 AM, Per Newgro  wrote:

> Hello,
>
> we are implementing a questionnaire tool for companies. I would like to
> import the data using a DIH.
>
> To increase performance i would like to use some caching. But my solution
> is not working. The score of my
>
> questionnaire is empty. But there is a value in the database. I've checked
> that.
>
> We can mark questionnaires for special purposes. I need to import the
> special mpc score. The mpc questionnaire
>
> is not changing while importing. So i thaught i can can cache this value
> for usage in mpc_score queries.
>
> Can you please help me, to find out what i'm doing wrong here?
>
> Thanks
>
> Per
>
> 
>  name="company"
> query="SELECT id as ID,
>   customer_number as CUSTOMER_NUMBER
> FROM  companies
>WHERE  '${dataimporter.request.clean}' != 'false'
>   OR  last_modified > '${dataimporter.last_index_tim
> e}'">
>  name="mpc"
> processor="SqlEntityProcessor"
> cacheImpl="SortedMapBackedCache"
> query="select qp.questionnaire AS ID
>from questionnaire_purposes qp
>join purposes p ON qp.id = p.id
>where p.name = 'mpc';">
>  name="mpc_score"
> query="select c.score as SCORE
>FROM basfcdi.census c
>where c.company=${company.ID}
>and c.questionnaire = ${mpc.ID};">
> 
> 
> 
> 
>
> 
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Memory leak in Solr

[William Bell]

What do you mean by JVM level? Run Solr on different ports on the same
machine? If you have a 32 core box would you run 2,3,4 JVMs?

On Sun, Dec 4, 2016 at 8:46 PM, Jeff Wartes  wrote:

>
> Here’s an earlier post where I mentioned some GC investigation tools:
> https://mail-archives.apache.org/mod_mbox/lucene-solr-user/
> 201604.mbox/%3c8f8fa32d-ec0e-4352-86f7-4b2d8a906...@whitepages.com%3E
>
> In my experience, there are many aspects of the Solr/Lucene memory
> allocation model that scale with things other than documents returned.
> (such as cardinality, or simply index size) A single query on a large index
> might consume dozens of megabytes of heap to complete. But that heap should
> also be released quickly after the query finishes.
> The key characteristic of a memory leak is that the software is allocating
> memory that it cannot reclaim. If it’s a leak, you ought to be able to
> reproduce it at any query rate - have you tried this? A run with, say, half
> the rate, over twice the duration?
>
> I’m inclined to agree with others here, that although you’ve correctly
> attributed the cause to GC, it’s probably less an indication of a leak, and
> more an indication of simply allocating memory faster than it can be
> reclaimed, combined with the long pauses that are increasingly unavoidable
> as heap size goes up.
> Note that in the case of a CMS allocation failure, the fallback full-GC is
> *single threaded*, which means it’ll usually take considerably longer than
> a normal GC - even for a comparable amount of garbage.
>
> In addition to GC tuning, you can address these by sharding more, both at
> the core and jvm level.
>
>
> On 12/4/16, 3:46 PM, "Shawn Heisey"  wrote:
>
> On 12/3/2016 9:46 PM, S G wrote:
> > The symptom we see is that the java clients querying Solr see
> response
> > times in 10s of seconds (not milliseconds).
> 
> > Some numbers for the Solr Cloud:
> >
> > *Overall infrastructure:*
> > - Only one collection
> > - 16 VMs used
> > - 8 shards (1 leader and 1 replica per shard - each core on separate
> VM)
> >
> > *Overview from one core:*
> > - Num Docs:193,623,388
> > - Max Doc:230,577,696
> > - Heap Memory Usage:231,217,880
> > - Deleted Docs:36,954,308
> > - Version:2,357,757
> > - Segment Count:37
>
> The heap memory usage number isn't useful.  It doesn't cover all the
> memory used.
>
> > *Stats from QueryHandler/select*
> > - requests:78,557
> > - errors:358
> > - timeouts:0
> > - totalTime:1,639,975.27
> > - avgRequestsPerSecond:2.62
> > - 5minRateReqsPerSecond:1.39
> > - 15minRateReqsPerSecond:1.64
> > - avgTimePerRequest:20.87
> > - medianRequestTime:0.70
> > - 75thPcRequestTime:1.11
> > - 95thPcRequestTime:191.76
>
> These times are in *milliseconds*, not seconds .. and these are even
> better numbers than you showed before.  Where are you seeing 10 plus
> second query times?  Solr is not showing numbers like that.
>
> If your VM host has 16 VMs on it and each one has a total memory size
> of
> 92GB, then if that machine doesn't have 1.5 terabytes of memory, you're
> oversubscribed, and this is going to lead to terrible performance...
> but
> the numbers you've shown here do not show terrible performance.
>
> > Plus, on every server, we are seeing lots of exceptions.
> > For example:
> >
> > Between 8:06:55 PM and 8:21:36 PM, exceptions are:
> >
> > 1) Request says it is coming from leader, but we are the leader:
> > update.distrib=FROMLEADER=HOSTB_ca_1_
> 1456430020/=javabin=2
> >
> > 2) org.apache.solr.common.SolrException: Request says it is coming
> from
> > leader, but we are the leader
> >
> > 3) org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: Tried one server
> for read
> > operation and it timed out, so failing fast
> >
> > 4) null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: Tried one server
> for read
> > operation and it timed out, so failing fast
> >
> > 5) org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: Tried one server
> for read
> > operation and it timed out, so failing fast
> >
> > 6) null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: Tried one server
> for read
> > operation and it timed out, so failing fast
> >
> > 7) org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers
> > available to handle this request. Zombie server list:
> > [HOSTA_ca_1_1456429897]
> >
> > 8) null:org.apache.solr.common.SolrException:
> > org.apache.solr.client.solrj.SolrServerException: No live
> SolrServers
> > 

timeAllowed?

[William Bell]

When should one use timeAllowed in Solr 5.5.3?

How much overhead is involved in it? Does it create a thread and a timer
event if you use it? Or is the timer event used even if you don't set it
explicitly?

We are thinking about using it for all requests. Just set it +10
milliseconds longer than our web requester timeout? Would that create a lot
of overhead?

Also what is the minimum value - what is the tradeoff? Setting it to 100
seems excessive, but might make sense on an autosuggest component?

Also does it include the time to stream the result? So if the payload was
super large, would it expire that?

The other option is to set it to only requests > 5 seconds... ?



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue in 5.5.3 with lucene localParams with type

[William Bell]

Bump...

Thoughts?

It seems that {!lucene type=} would just override the lucene qp -
but in 5.5.3 something changed.

On Mon, Nov 28, 2016 at 1:15 PM, William Bell <billnb...@gmail.com> wrote:

> In Solr 5.4.1 this used to work:
>
> fl={!lucene%20type=payloadQueryParser v='hosp_quality_spec_boost:PS628'}
>
> 24.227154,
>
> The only way I can get payloads to work is:
>
> fl={!payloadQueryParser v='hosp_quality_spec_boost:PS628'}
>
> 0.125,
>
> But the right values only come back in #2. It should be .125.
>
> Why is type not working anymore for queryParser?
>
> 
>  class="hg.payload.PayloadQParserPlugin"/>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Issue in 5.5.3 with lucene localParams with type

[William Bell]

In Solr 5.4.1 this used to work:

fl={!lucene%20type=payloadQueryParser v='hosp_quality_spec_boost:PS628'}

24.227154,

The only way I can get payloads to work is:

fl={!payloadQueryParser v='hosp_quality_spec_boost:PS628'}

0.125,

But the right values only come back in #2. It should be .125.

Why is type not working anymore for queryParser?





-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Aggregate Values Inside a Facet Range

[William Bell]

How about hours?

NOW+1HR
NOW+2HR
NOW+12HR
NOW-4HR

Can we add that?


On Fri, Nov 4, 2016 at 12:25 PM, Furkan KAMACI 
wrote:

> I have documents like that
>
> id:5
> timestamp:NOW //pseudo date representation
> count:13
>
> id:4
> timestamp:NOW //pseudo date representation
> count:3
>
> id:3
> timestamp:NOW-1DAY //pseudo date representation
> count:21
>
> id:2
> timestamp:NOW-1DAY //pseudo date representation
> count:29
>
> id:1
> timestamp:NOW-3DAY //pseudo date representation
> count:4
>
> When I want to facet last 3 days data by timestamp its OK. However my need
> is that:
>
> facets:
> TODAY: 16 //pseudo representation
> TODAY - 1: 50 //pseudo date representation
> TODAY - 2: 0 //pseudo date representation
> TODAY - 3: 4 //pseudo date representation
>
> I mean, I have to facet by dates and aggregate values inside that facet
> range. Is it possible to do that without multiple queries at Solr?
>
> Kind Regards,
> Furkan KAMACI
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Problem with Password Decryption in Data Import Handler

[William Bell]

OK it was

echo -n "${encrypt_key}" > encrypt.key



On Thu, Nov 3, 2016 at 12:20 PM, William Bell <billnb...@gmail.com> wrote:

> I cannot get it to work either.
>
> Here are my steps. I took the key from the Patch in
> https://issues.apache.org/jira/secure/attachment/12730862/SOLR-4392.patch.
>
> echo U2FsdGVkX19Gz7q7/4jj3Wsin7801TlFbob1PBT2YEacbPE
> UARDiuV5zGSAwU4Sz7upXDEPIQPU48oY1fBWM6Q== > pass.enc
>
> openssl aes-128-cbc -d -a -salt -in pass.enc
>
> I typed: Password
>
> enter aes-128-cbc decryption password:
>
> SomeRandomEncryptedTextUsingAES128
>
> I cannot find a test case in the latest v5.5.3 code.? It seems like
> openssl command is wrong?
>
> So it worked for that. Not sure if the code changed, but after doing this
> I get in solr.log:
>
>
> 2016-11-03 12:06:20.139 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.p.LogUpdateProcessorFactory [autosuggestfull]  webapp=/solr
> path=/dataimport params={debug=false=false=true=
> false=false=json=full-import=spec=false}
> status=0 QTime=19{} 0 64
>
> 2016-11-03 12:06:20.140 ERROR (Thread-127) [   x:autosuggestfull]
> o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException:
> java.lang.RuntimeException: 
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:270)
>
> at org.apache.solr.handler.dataimport.DataImporter.
> doFullImport(DataImporter.java:416)
>
> at org.apache.solr.handler.dataimport.DataImporter.
> runCmd(DataImporter.java:480)
>
> at org.apache.solr.handler.dataimport.DataImporter$1.run(
> DataImporter.java:461)
>
> Caused by: java.lang.RuntimeException: 
> org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:416)
>
> at org.apache.solr.handler.dataimport.DocBuilder.
> doFullDump(DocBuilder.java:329)
>
> at org.apache.solr.handler.dataimport.DocBuilder.execute(
> DocBuilder.java:232)
>
> ... 3 more
>
> Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
> Error decoding password Processing Document # 1
>
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> decryptPwd(JdbcDataSource.java:131)
>
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> init(JdbcDataSource.java:74)
>
> at org.apache.solr.handler.dataimport.DataImporter.
> getDataSourceInstance(DataImporter.java:389)
>
> at org.apache.solr.handler.dataimport.ContextImpl.
> getDataSource(ContextImpl.java:100)
>
> at org.apache.solr.handler.dataimport.SqlEntityProcessor.
> init(SqlEntityProcessor.java:53)
>
> at org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(
> EntityProcessorWrapper.java:75)
>
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:433)
>
> at org.apache.solr.handler.dataimport.DocBuilder.
> buildDocument(DocBuilder.java:414)
>
> ... 5 more
>
> Caused by: java.lang.IllegalStateException: Bad password, algorithm, mode
> or padding; no salt, wrong number of iterations or corrupted ciphertext.
>
> at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:249)
>
> at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:195)
>
> at org.apache.solr.handler.dataimport.JdbcDataSource.
> decryptPwd(JdbcDataSource.java:129)
>
> ... 12 more
>
> Caused by: javax.crypto.BadPaddingException: Given final block not
> properly padded
>
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:975)
>
> at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:833)
>
> at com.sun.crypto.provider.AESCipher.engineDoFinal(
> AESCipher.java:446)
>
> at javax.crypto.Cipher.doFinal(Cipher.java:2165)
>
> at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:245)
>
> ... 14 more
>
>
> 2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.DirectUpdateHandler2 start rollback{}
>
> 2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter...
> core=autosuggestfull
>
> 2016-11-03 12:06:20.154 INFO  (Thread-127) [   x:autosuggestfull]
> o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1
>
>
>
> On Wed, Nov 2, 2016 at 12:21 PM, Jamie Jackson <jamieja...@gmail.com>

Re: Problem with Password Decryption in Data Import Handler

[William Bell]

I cannot get it to work either.

Here are my steps. I took the key from the Patch in
https://issues.apache.org/jira/secure/attachment/12730862/SOLR-4392.patch.

echo
U2FsdGVkX19Gz7q7/4jj3Wsin7801TlFbob1PBT2YEacbPEUARDiuV5zGSAwU4Sz7upXDEPIQPU48oY1fBWM6Q==
> pass.enc

openssl aes-128-cbc -d -a -salt -in pass.enc

I typed: Password

enter aes-128-cbc decryption password:

SomeRandomEncryptedTextUsingAES128

I cannot find a test case in the latest v5.5.3 code.? It seems like openssl
command is wrong?

So it worked for that. Not sure if the code changed, but after doing this I
get in solr.log:


2016-11-03 12:06:20.139 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.p.LogUpdateProcessorFactory [autosuggestfull]  webapp=/solr
path=/dataimport
params={debug=false=false=true=false=false=json=full-import=spec=false}
status=0 QTime=19{} 0 64

2016-11-03 12:06:20.140 ERROR (Thread-127) [   x:autosuggestfull]
o.a.s.h.d.DataImporter Full Import failed:java.lang.RuntimeException:
java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Error
decoding password Processing Document # 1

at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:270)

at
org.apache.solr.handler.dataimport.DataImporter.doFullImport(DataImporter.java:416)

at
org.apache.solr.handler.dataimport.DataImporter.runCmd(DataImporter.java:480)

at
org.apache.solr.handler.dataimport.DataImporter$1.run(DataImporter.java:461)

Caused by: java.lang.RuntimeException:
org.apache.solr.handler.dataimport.DataImportHandlerException: Error
decoding password Processing Document # 1

at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:416)

at
org.apache.solr.handler.dataimport.DocBuilder.doFullDump(DocBuilder.java:329)

at
org.apache.solr.handler.dataimport.DocBuilder.execute(DocBuilder.java:232)

... 3 more

Caused by: org.apache.solr.handler.dataimport.DataImportHandlerException:
Error decoding password Processing Document # 1

at
org.apache.solr.handler.dataimport.JdbcDataSource.decryptPwd(JdbcDataSource.java:131)

at
org.apache.solr.handler.dataimport.JdbcDataSource.init(JdbcDataSource.java:74)

at
org.apache.solr.handler.dataimport.DataImporter.getDataSourceInstance(DataImporter.java:389)

at
org.apache.solr.handler.dataimport.ContextImpl.getDataSource(ContextImpl.java:100)

at
org.apache.solr.handler.dataimport.SqlEntityProcessor.init(SqlEntityProcessor.java:53)

at
org.apache.solr.handler.dataimport.EntityProcessorWrapper.init(EntityProcessorWrapper.java:75)

at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:433)

at
org.apache.solr.handler.dataimport.DocBuilder.buildDocument(DocBuilder.java:414)

... 5 more

Caused by: java.lang.IllegalStateException: Bad password, algorithm, mode
or padding; no salt, wrong number of iterations or corrupted ciphertext.

at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:249)

at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:195)

at
org.apache.solr.handler.dataimport.JdbcDataSource.decryptPwd(JdbcDataSource.java:129)

... 12 more

Caused by: javax.crypto.BadPaddingException: Given final block not properly
padded

at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:975)

at com.sun.crypto.provider.CipherCore.doFinal(CipherCore.java:833)

at
com.sun.crypto.provider.AESCipher.engineDoFinal(AESCipher.java:446)

at javax.crypto.Cipher.doFinal(Cipher.java:2165)

at org.apache.solr.util.CryptoKeys.decodeAES(CryptoKeys.java:245)

... 14 more


2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.DirectUpdateHandler2 start rollback{}

2016-11-03 12:06:20.140 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.u.DefaultSolrCoreState Rollback old IndexWriter...
core=autosuggestfull

2016-11-03 12:06:20.154 INFO  (Thread-127) [   x:autosuggestfull]
o.a.s.c.SolrDeletionPolicy SolrDeletionPolicy.onInit: commits: num=1



On Wed, Nov 2, 2016 at 12:21 PM, Jamie Jackson  wrote:

> I'm at a brick wall. Here's the latest status:
>
> Here are some sample commands that I'm using:
>
> *Create the encryptKeyFile and encrypted password:*
>
>
> encrypter_password='this_is_my_encrypter_password'
> plain_db_pw='Oakton153'
>
> cd /var/docker/solr_stage2/credentials/
> echo -n "${encrypter_password}" > encpwd.txt
> echo -n "${plain_db_pwd}" > plaindbpwd.txt
> openssl enc -aes-128-cbc -a -salt -in plaindbpwd.txt -k
> "${encrypter_password}"
>
> rm plaindbpwd.txt
>
> That generated this as the password, by the way:
>
> U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o=
>
> *Configure DIH configuration:*
>
> 
>
>  driver="org.mariadb.jdbc.Driver"
> url="jdbc:mysql://local.mysite.com:3306/mysite"
> user="root"
> password="U2FsdGVkX19pBVTeZaSl43gFFAlrx+Th1zSg1GvlX9o="
> 

Re: OOM Error

[William Bell]

I would also recommend that 8GB is cutting it close for Java 8 JVM with
SOLR. We use 12GB and have had issues with 8GB. But your mileage may vary.

On Tue, Oct 25, 2016 at 1:37 AM, Toke Eskildsen 
wrote:

> On Mon, 2016-10-24 at 18:27 -0400, Susheel Kumar wrote:
> > I am seeing OOM script killed solr (solr 6.0.0) on couple of our VM's
> > today. So far our solr cluster has been running fine but suddenly
> > today many of the VM's Solr instance got killed.
>
> As you have the GC-logs, you should be able to determine if it was a
> slow death (e.g. caches gradually being filled) or a sudden one (e.g.
> grouping or faceting on a large new non-DocValued field).
>
> Try plotting the GC logs with time on the x-axis and free memory after
> GC on the y-axis. It it happens to be a sudden death, the last lines in
> solr.log might hold a clue after all.
>
> - Toke Eskildsen, State and University Library, Denmark
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Migrating to Solr 6.1.0 from 5.5.0

[William Bell]

the documentation is not good on this. Not sure how to fix it either.

On Tue, Sep 27, 2016 at 3:41 AM, M, Arjun (Nokia - IN/Bangalore) <
arju...@nokia.com> wrote:

> Hi,
>
> We are getting the below errors when migrating Solr from 5.5.0 to
> 6.1.0. Could anyone help in resolving the issue, if you have come across
> this?
>
> org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException:
> Error from server at http://127.0.0.1:41569/solr/collection1: Unable to
> parse shape given formats "lat,lon", "x y" or as WKT because
> java.text.ParseException: java.lang.UnsupportedOperationException:
> Unsupported shape of this SpatialContext. Try JTS or Geo3D. input:
> POLYGON((-10 30, -40 40, -10 -20, 0 0, -10 30))
>
> Thanks in advance..
>
> Thanks & Regards,
>Arjun M
>
>
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Search with the start of field

[William Bell]

Show us the FieldType and Field definitions.

On Wed, Sep 21, 2016 at 12:06 AM, Mahmoud Almokadem 
wrote:

> Hello,
>
> What is the best way to search with the start token of field?
>
> For example: the field contains these values
>
> Document1: ABC  DEF GHI
> Document2: DEF GHI JKL
>
> when I search with DEF, I want to get Document2 only. Is that possible?
>
> Thanks,
> Mahmoud
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Healthgrades looking for Search Product Manager

[William Bell]

Hi all.

I am reaching out to see if anyone of the Search experts in this group
would consider helping us out on Healthgrades search. We have a ton of
ideas from chat bots, to personalization, collaborative filtering, and are
going to build some ridiculously crazy ideas that consumers use to find a
doctor.

We are building a world class search experience and need help doing it.

Looking for FTE - can be anywhere but prefer Denver, CO or Consultant.

If interested reach out to me (bb...@healthgrades.com).

Thanks.

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: changed query parsing between 4.10.4 and 5.5.3?

[William Bell]

We need to get this fixed.

Do you have some sample queries?

On Wed, Sep 14, 2016 at 12:37 AM, Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

> Hi Greg,
>
> after trying several hours with all combinations of parameters and not
> getting any useful search result with complex search terms and edismax
> I finally copied o.a.s.s.ExtendedDismaxQParser.java from version 4.10.4
> to 5.5.3 and did a little modification in o.a.s.u.SolrPluginUtils.java.
>
> Now it is searching correct and getting logical and valid search results
> with any kind of complex search.
> Problem solved.
>
> But still, the edismax, at least of 5.5.3, has some bugs.
> If I get time I will look into this but right now my problem is solved
> and the customers and users are happy.
>
> I hope that this buggy edismax version is not used in solr 6.x otherwise
> you
> have the same problems there.
>
> Regards
> Bernd
>
>
> Am 12.09.2016 um 05:10 schrieb Greg Pendlebury:
> > Hi Bernd,
> >
> > "From my point of view the old parsing behavior was correct.
> > If searching for a term without operator it is always OR, otherwise
> > you can add "+" or "-" to modify that. Now with q.op AND it is
> > modified to "+" as a MUST."
> >
> > It is correct in both cases. q.op dictates (for that query) what default
> > operator to use when none is provided, and it is used as a priority over
> > the system whole 'defaultOperator'. In either case, if you ask it to use
> > OR, it uses it; if you ask it to use AND, it uses it. The behaviour from
> > 4.10 that was changed (arguably fixed, although I know that is a
> debatable
> > point) was that you asked it to use AND, and it ignored you (irrespective
> > of whether you used defaultOperator or q.op). The are a few subtle
> > distinctions that are being missed (like the difference between the
> boolean
> > operators and the OCCURS flags that your are talking about), but they are
> > not going to change the outcome.
> >
> > 8812 related to users who had been historically setting the q.op
> parameter
> > to influence the downstream default selection of 'mm' (If you don't
> provide
> > 'mm' it is set for you based on 'q.op') instead of directly setting the
> > 'mm' value themselves. But again in this case, you're setting 'mm'
> anyway,
> > so it shouldn't be relevant.
> >
> > Ta,
> > Greg
> >
> > On 9 September 2016 at 16:44, Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de>
> > wrote:
> >
> >> Hi Greg,
> >>
> >> thanks a lot, thats it.
> >> After setting q.op to OR it works _nearly_ as before with 4.10.4.
> >>
> >> But how stupid this?
> >> I have in my schema 
> >> and also had q.op to AND to make sure my default _is_ AND,
> >> meant as conjunction between terms.
> >> But now I have q.op to OR and defaultOperator in schema to AND
> >> to just get _nearly_ my old behavior back.
> >>
> >> schema has following comment:
> >> "... The default is OR, which is generally assumed so it is
> >> not a good idea to change it globally here.  The "q.op" request
> >> parameter takes precedence over this. ..."
> >>
> >> What I don't understand is why they change some major internals
> >> and don't give any notice about how to keep old parsing behavior.
> >>
> >> From my point of view the old parsing behavior was correct.
> >> If searching for a term without operator it is always OR, otherwise
> >> you can add "+" or "-" to modify that. Now with q.op AND it is
> >> modified to "+" as a MUST.
> >>
> >> I still get some differences in search results between 4.10.4 and 5.5.3.
> >> What other side effects has this change of q.op from AND to OR in
> >> other parts of query handling, parsing and searching?
> >>
> >> Regards
> >> Bernd
> >>
> >> Am 09.09.2016 um 05:43 schrieb Greg Pendlebury:
> >>> I forgot to mention the tickets:
> >>> SOLR-2649 and SOLR-8812
> >>>
> >>> On 9 September 2016 at 13:38, Greg Pendlebury <
> greg.pendleb...@gmail.com
> >>>
> >>> wrote:
> >>>
>  Under 4.10 q.op was ignored by the edismax parser and always forced to
> >> OR.
>  5.5 is looking at the q.op=AND you requested.
> 
>  There are also some changes to the default values selected for mm,
> but I
>  doubt those apply here since you are setting it explicitly.
> 
>  On 8 September 2016 at 00:35, Mikhail Khludnev 
> wrote:
> 
> > I suppose
> >+((text:star text:trek)~2)
> > and
> >   +(+text:star +text:trek)
> > are equal. mm=2 is equal to +foo +bar
> >
> > On Wed, Sep 7, 2016 at 10:52 AM, Bernd Fehling <
> > bernd.fehl...@uni-bielefeld.de> wrote:
> >
> >> Hi list,
> >>
> >> while going from SOLR 4.10.4 to 5.5.3 I noticed a change in query
> > parsing.
> >> 4.10.4
> >> text:star text:trek
> >>   text:star text:trek
> >>   (+((text:star
> text:trek)~2))/no_coord
> >>   +((text:star text:trek)~2)
> >>
> >> 5.5.3
> >> text:star text:trek
> >>   text:star text:trek
> >>   (+(+text:star +text:trek))/no_coord
> 

Re: Multilevel grouping?

[William Bell]

On grouping, if I only want the first 5 responses, it would be great if the
code short-circuited to improve performance.

I am not sure I want it grouping 10M results, when I already have 5 that
are good enough. ?

On Thu, Jul 14, 2016 at 10:33 AM, Callum Lamb  wrote:

> Look at the collapse module
>
> https://cwiki.apache.org/confluence/display/solr/Collapse+and+Expand+Results
> .
> It can the same thing as group.
>
> If you want to get counts/facets from before the collapse, tag the collapse
> statement use the exclude tags tags in your json facets (there's an
> equivalent for non json facets). I think the default nullpolicy is
> different from grouping too, but you can change it to be the same.
>
> I've not been able to get 2 collapses to work on my version of Solr. But
> collapse + group works and you can get 2 levels. Not being able to do
> multiple collapses appears to be a bug (it sorta works). I recall there
> being JIRA case somewhere stating it was fixed in some version. So you may
> be able to do as many levels as you like if you upgrade/already run a very
> recent version of Solr.
>
>
>
>
>
> On Thu, Jul 14, 2016 at 3:52 PM, Aditya Sundaram <
> aditya.sunda...@myntra.com
> > wrote:
>
> > Thanks Yonik, was looking for exactly that, is there any workaround to
> > achieve that currently?
> >
> > On Tue, Jul 12, 2016 at 5:07 PM, Yonik Seeley  wrote:
> >
> > > I started this a while ago, but haven't found the time to finish:
> > > https://issues.apache.org/jira/browse/SOLR-7830
> > >
> > > -Yonik
> > >
> > >
> > > On Tue, Jul 12, 2016 at 7:29 AM, Aditya Sundaram
> > >  wrote:
> > > > Does solr support multilevel grouping? I want to group upto 2/3
> levels
> > > > based on different fields i.e 1st group on field one, within which i
> > > group
> > > > by field 2 etc.
> > > > I am aware of facet.pivot which does the same but retrieves only the
> > > count.
> > > > Is there anyway to get the documents as well along with the count in
> > > > facet.pivot???
> > > >
> > > > --
> > > > Aditya Sundaram
> > >
> >
> >
> >
> > --
> > Aditya Sundaram
> > Software Engineer, Technology team
> > AKR Tech park B Block, B1 047
> > +91-9844006866
> >
>
> --
>
> Mintel Group Ltd | 11 Pilgrim Street | London | EC4V 6RN
> Registered in England: Number 1475918. | VAT Number: GB 232 9342 72
>
> Contact details for our other offices can be found at
> http://www.mintel.com/office-locations.
>
> This email and any attachments may include content that is confidential,
> privileged
> or otherwise protected under applicable law. Unauthorised disclosure,
> copying, distribution
> or use of the contents is prohibited and may be unlawful. If you have
> received this email in error,
> including without appropriate authorisation, then please reply to the
> sender about the error
> and delete this email and any attachments.
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Automatic Language Identification

[William Bell]

We should add a simple filter in Solr for this. The current way requires
indexing.

https://github.com/kkrugler/yalder is good, it would be a great filter:

if NOT english, fail the whole text.

On Fri, Jul 1, 2016 at 6:33 AM, Allison, Timothy B. 
wrote:

> +1 to langdetect
>
> In Tika 2.0, we're going to remove our own language detection code and
> allow users to select Optimaize (fork of langdetect), MIT Lincoln Lab’s
> Text.jl library or Yalder (https://github.com/kkrugler/yalder).  The
> first two are now available in Tika 1.13.
>
> -Original Message-
> From: Markus Jelsma [mailto:markus.jel...@openindex.io]
> Sent: Wednesday, June 22, 2016 8:27 AM
> To: solr-user@lucene.apache.org; solr-user 
> Subject: RE: Automatic Language Identification
>
> Hello,
>
> I recommend using the langdetect language detector, it supports many more
> languages and has much higher precission than Tika's detector.
>
> Markus
>
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: integrate SOLR with OSM

[William Bell]

OSM is literally Postgres data. If you want to move some of this data into
SOLR, you can write SQL and import it...

On Sun, Jun 26, 2016 at 10:32 AM, Youzha <yuza.ras...@gmail.com> wrote:

> does anybody can help me for this case pls?
> On Jun 26, 2016 3:38 AM, "tkg_cangkul" <yuza.ras...@gmail.com> wrote:
>
> > hi william,
> >
> > thx for your reply.
> > is it any link article or documents that i can read for this?
> > pls let me know if you've any suggestion.
> >
> > On 26/06/16 02:58, William Bell wrote:
> >
> >> It depends on if your are using TIGERLINE data in OSM to increase the
> >> accuracy of home addresses.
> >>
> >> If you are using TIGERLINE, you are talking about > 500M rows or so +
> >> millions of OSM data. Which will is a LOT of data that you will be
> pumping
> >> into SOLR. You will need to use Solr Cloud since you will hit limits...
> >>
> >> With that amount of data I had issues with just USA OSM data with
> >> performance.
> >>
> >> If you get it working, please share.
> >>
> >> On Sat, Jun 25, 2016 at 12:49 PM, tkg_cangkul <yuza.ras...@gmail.com>
> >> wrote:
> >>
> >> hi i wanna try to integrate SOLR with OpenStreetMap (OSM). well the plan
> >>> is i wanna index some cordinaate (long & lat) to SOLR and then the OSM
> >>> will
> >>> try to showing the map of that coordinate. is there any article about
> >>> that?
> >>> pls help. i'm still confuse about this.
> >>>
> >>> thx before.
> >>>
> >>>
> >>
> >>
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: integrate SOLR with OSM

[William Bell]

It depends on if your are using TIGERLINE data in OSM to increase the
accuracy of home addresses.

If you are using TIGERLINE, you are talking about > 500M rows or so +
millions of OSM data. Which will is a LOT of data that you will be pumping
into SOLR. You will need to use Solr Cloud since you will hit limits...

With that amount of data I had issues with just USA OSM data with
performance.

If you get it working, please share.

On Sat, Jun 25, 2016 at 12:49 PM, tkg_cangkul  wrote:

> hi i wanna try to integrate SOLR with OpenStreetMap (OSM). well the plan
> is i wanna index some cordinaate (long & lat) to SOLR and then the OSM will
> try to showing the map of that coordinate. is there any article about that?
> pls help. i'm still confuse about this.
>
> thx before.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: issue with 5.3.1 and index version

[William Bell]

What we are trying to do is - we upgraded the Solr Master machine to 5.4.1,
and the slave is 5.3.1. When it replicates to the slave, it says
"incompatible types". That seems like a valid use case to me?

BIll


On Thu, Mar 31, 2016 at 2:49 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> The Lucene54 is the codec name. The luceneVer in solrconfig.xml is the
> compatibility version which is used by some analyzers/tokenizers/token
> filters to provide defaults or behaviour compatible with older versions. It
> has no relation to the indexing codec being used and once an index has been
> written to by a newer version of Lucene, going back to an old version is
> not possible in most cases.
>
> On Thu, Mar 31, 2016 at 4:00 AM, William Bell <billnb...@gmail.com> wrote:
>
> > When I index 5.4.1 using luceneVer in solrlconfig.xml of 5.3.1, the
> > segmentsw_9 files has in it Lucene54. Why? Is this a known bug?
> >
> > #strings segments_9
> >
> > segments
> >
> > Lucene54
> >
> > commitTimeMSec
> >
> > 1459374733276
> >
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

issue with 5.3.1 and index version

[William Bell]

When I index 5.4.1 using luceneVer in solrlconfig.xml of 5.3.1, the
segmentsw_9 files has in it Lucene54. Why? Is this a known bug?

#strings segments_9

segments

Lucene54

commitTimeMSec

1459374733276




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Example Transformer

[William Bell]

I wrote a transformer for DIH to get the value from a field using Http call.
Since this runs on a SOLR node when indexing, I think I would do it with
SOLRJ.

How do I take this and convert it to SOLRJ and avoid the network call ?

Also, this is pretty cool, and avoids the Entity for the call - since using
an Entity has issues which I reported on a few days ago. It is much easier
to access all fields when using a transformer.

http://localhost:8983/solr/survey/select;
qt="dihsurvey" source="provider_json" />

package hg;

import net.sf.json.JSON;
import net.sf.json.JSONObject;
import net.sf.json.JSONArray;
import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.Reader;
import java.net.URL;
import java.net.URLConnection;
import java.nio.charset.Charset;

import java.io.IOException;
import org.apache.solr.handler.dataimport.Context;
import org.apache.solr.handler.dataimport.DataImporter;
import org.apache.solr.handler.dataimport.Transformer;

import java.util.List;
import java.util.Map;


// Overwrite for Survey embedded entity
// http://localhost:8983/solr/survey/select; qt="dihsurvey"
source="provider_json" />

public class JSONProviderSurvey extends Transformer {

private static String readAll(Reader rd) throws IOException {
StringBuilder sb = new StringBuilder();
int cp;
while ((cp = rd.read()) != -1) {
sb.append((char) cp);
}
return sb.toString();
}

public static JSONObject readJsonFromUrl(String urlPath) throws
IOException {
URL url = new URL(urlPath);
URLConnection con = url.openConnection();
con.setConnectTimeout(1);
con.setReadTimeout(1);
InputStream is = con.getInputStream();
//InputStream is = new URL(url).openStream();
try {
BufferedReader rd = new BufferedReader(new
InputStreamReader(is, Charset.forName("UTF-8")));
String jsonText = readAll(rd);
JSONObject json = JSONObject.fromObject(jsonText);
return json;
} finally {
is.close();
}
}

public Map transformRow(Map row,
Context context) {

List> fields = context.getAllEntityFields();

for (Map field : fields) {
String jsonf = field.get("jsonprovidersurvey");

if ("true".equals(jsonf)) {

String columnName = field.get(DataImporter.COLUMN);
String url = field.get("url");
//url = "http://hgsolr2devmstr:8983/solr/survey/select;;
String qt = field.get("qt");
//qt = "dihsurvey";
String pwid = SolrUtility.GetSafeString(row.get("id"));
String source = field.get("source");
String outfield =
SolrUtility.GetSafeString(field.get("outfield"));
if (row.get(source) != null) {

Object jobject = row.get(source);
JSONObject provider =
JSONObject.fromObject(jobject.toString());
JSONObject json;
try {
json = readJsonFromUrl(url + "?qt=" + qt +
"=provider_code:" + pwid.toUpperCase() +
"=json=1=true");
if (json.get("response") != null) {
JSONObject response =
json.getJSONObject("response");
JSONArray docs = response.getJSONArray("docs");
if (docs.size() > 0) {
JSONObject doc = docs.getJSONObject(0);
boolean has_comment =
SolrUtility.GetSafeBool(doc.get("has_comment"));
provider.element("HasOpenComment", has_comment);
row.put(columnName, provider.toString());
if (outfield.length() > 0) {
row.put(outfield, has_comment);
}
}
}
} catch (Exception e) {
provider.element("HasOpenComment", false);
row.put(columnName, provider.toString());
if (outfield.length() > 0) {
row.put(outfield, false);
}
}
}
}
}

return row;
}
}



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

DIG issue with SolrEntityProcessor 5.4.1

[William Bell]

We are running this inside of another entity in DIH. There appears to be an
issue. We get 2 calls to the survey core if hits > 0. If hits = 0 we get 1
call. Has anyone else seen this? Shall I fix it? Any ideas where this bug
may be?

http://localhost:8983/solr/survey; qt="dihsurvey" query="provider_code:${
item.id}" rows="1">  


2016-03-16 09:57:11.349 INFO  (qtp796684896-71) [   x:survey]
o.a.s.c.S.Request [survey] webapp=/solr path=/select
params={q=provider_code:X4L26=dihsurvey=0=30=1=javabin=2}
hits=0 status=0 QTime=1

2016-03-16 09:57:11.398 INFO  (qtp796684896-23) [   x:survey]
o.a.s.c.S.Request [survey] webapp=/solr path=/select
params={q=provider_code:2J8CP=dihsurvey=0=30=1=javabin=2}
hits=427 status=0 QTime=2

2016-03-16 09:57:11.402 INFO  (qtp796684896-173) [   x:survey]
o.a.s.c.S.Request [survey] webapp=/solr path=/select
params={q=provider_code:2J8CP=dihsurvey=1=30=1=javabin=2}
hits=427 status=0 QTime=1

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: DIG issue with SolrEntityProcessor 5.4.1

[William Bell]

I will try to see if I can create a use case and fix it.

On Wed, Mar 16, 2016 at 10:00 AM, William Bell <billnb...@gmail.com> wrote:

> We are running this inside of another entity in DIH. There appears to be
> an issue. We get 2 calls to the survey core if hits > 0. If hits = 0 we get
> 1 call. Has anyone else seen this? Shall I fix it? Any ideas where this bug
> may be?
>
> http://localhost:8983/solr/survey; qt="dihsurvey" query="provider_code:${
> item.id}" rows="1">   entity>
>
>
> 2016-03-16 09:57:11.349 INFO  (qtp796684896-71) [   x:survey]
> o.a.s.c.S.Request [survey] webapp=/solr path=/select
> params={q=provider_code:X4L26=dihsurvey=0=30=1=javabin=2}
> hits=0 status=0 QTime=1
>
> 2016-03-16 09:57:11.398 INFO  (qtp796684896-23) [   x:survey]
> o.a.s.c.S.Request [survey] webapp=/solr path=/select
> params={q=provider_code:2J8CP=dihsurvey=0=30=1=javabin=2}
> hits=427 status=0 QTime=2
>
> 2016-03-16 09:57:11.402 INFO  (qtp796684896-173) [   x:survey]
> o.a.s.c.S.Request [survey] webapp=/solr path=/select
> params={q=provider_code:2J8CP=dihsurvey=1=30=1=javabin=2}
> hits=427 status=0 QTime=1
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Field exclusion from fl and hl.fl

[William Bell]

Can we get this over the goal line?

https://issues.apache.org/jira/browse/SOLR-3191

On Sun, Mar 6, 2016 at 3:16 AM, Zheng Lin Edwin Yeo <edwinye...@gmail.com>
wrote:

> Hi,
>
> No, I tried that and I got the following error.
>
> {
>   "responseHeader":{
> "status":500,
> "QTime":0},
>   "error":{
> "msg":"For input string: \"-\"",
> "trace":"java.lang.NumberFormatException: For input string:
> \"-\"\r\n\tat
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\r\n\tat
> java.lang.Long.parseLong(Long.java:581)\r\n\tat
> java.lang.Long.parseLong(Long.java:631)\r\n\tat
> org.apache.solr.search.StrParser.getNumber(StrParser.java:124)\r\n\tat
>
> org.apache.solr.search.FunctionQParser.parseValueSource(FunctionQParser.java:298)\r\n\tat
>
> org.apache.solr.search.FunctionQParser.parse(FunctionQParser.java:80)\r\n\tat
> org.apache.solr.search.QParser.getQuery(QParser.java:141)\r\n\tat
>
> org.apache.solr.search.SolrReturnFields.add(SolrReturnFields.java:297)\r\n\tat
>
> org.apache.solr.search.SolrReturnFields.parseFieldList(SolrReturnFields.java:113)\r\n\tat
>
> org.apache.solr.search.SolrReturnFields.(SolrReturnFields.java:99)\r\n\tat
>
> org.apache.solr.search.SolrReturnFields.(SolrReturnFields.java:75)\r\n\tat
>
> org.apache.solr.handler.component.QueryComponent.prepare(QueryComponent.java:139)\r\n\tat
>
> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:247)\r\n\tat
>
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:156)\r\n\tat
> org.apache.solr.core.SolrCore.execute(SolrCore.java:2073)\r\n\tat
> org.apache.solr.servlet.HttpSolrCall.execute(HttpSolrCall.java:658)\r\n\tat
> org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:457)\r\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:222)\r\n\tat
>
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:181)\r\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)\r\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)\r\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)\r\n\tat
>
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:577)\r\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)\r\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)\r\n\tat
>
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)\r\n\tat
>
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)\r\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)\r\n\tat
>
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)\r\n\tat
>
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:215)\r\n\tat
>
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)\r\n\tat
>
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)\r\n\tat
> org.eclipse.jetty.server.Server.handle(Server.java:499)\r\n\tat
> org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:310)\r\n\tat
>
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)\r\n\tat
>
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:540)\r\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)\r\n\tat
>
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)\r\n\tat
> java.lang.Thread.run(Thread.java:745)\r\n",
> "code":500}}
>
>
> Regards,
> Edwin
>
>
> On 6 March 2016 at 11:19, William Bell <billnb...@gmail.com> wrote:
>
> > it used to support
> >
> > fl=*,-field
> >
> > Does that not work now?
> >
> > On Sat, Mar 5, 2016 at 7:37 PM, Zheng Lin Edwin Yeo <
> edwinye...@gmail.com>
> > wrote:
> >
> > > I have yet to find any workaround so far.Still have to list out all the
> > > remaining fields one by one.
> > >
> > > Does anyone else has any suggestions?
> > >
> > > Regards,
> > > Edwin
> > >
> > >
> > > On 18 February 2016 at 17:07, Anil <anilk...@gmail.com> wrote:
> > >
> > > > I am looking for the same. please do let me know just in case you
> find
> > > > workaro

Re: Field exclusion from fl and hl.fl

[William Bell]

it used to support

fl=*,-field

Does that not work now?

On Sat, Mar 5, 2016 at 7:37 PM, Zheng Lin Edwin Yeo 
wrote:

> I have yet to find any workaround so far.Still have to list out all the
> remaining fields one by one.
>
> Does anyone else has any suggestions?
>
> Regards,
> Edwin
>
>
> On 18 February 2016 at 17:07, Anil  wrote:
>
> > I am looking for the same. please do let me know just in case you find
> > workaround.
> >
> > On 18 February 2016 at 14:18, Zheng Lin Edwin Yeo 
> > wrote:
> >
> > > Hi,
> > >
> > > Would like to find out, is there already a way to exclude field from
> the
> > > Solr response. I did came across SOLR-3191 which was created about 4
> > years
> > > ago, but could not find any workable solutions from there.
> > >
> > > As my collections can have more than 100 fields, and I would need to
> > return
> > > the majority of then except for one or two, so if there is a way to
> > exclude
> > > the fields would be good, if not I have to list all the remaining
> fields
> > > (which can be more than 100 for each collections).
> > >
> > > I am using Solr 5.4.0.
> > >
> > > Regards,
> > > Edwin
> > >
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

XX:ParGCCardsPerStrideChunk

[William Bell]

Has anyone tried -XX:ParGCCardsPerStrideChunk with Solr?

There has been reports of improved GC times.

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

What search metrics are useful?

[William Bell]

How do others look at search metrics?

1. Search conversion? Do you look at searches and if the user does not
click on a result, and reruns the search that would be a failure?

2. How to measure auto complete success metrics?

3. Facets/filters could be considered negative, since we did not find the
results that the user wanted, and now they are filtering - who to measure?

4. One easy metric is searches with 0 results. We could auto expand the geo
distance or ask the user "did you mean" ?

5. Another easy one would be tech performance: "time it takes in seconds to
get a result".

6. How to measure fuzzy? How do you know you need more synonyms? How to
measure?

7. How many searches it takes before the user clicks on a result?

Other ideas? Is there a video or presentation on search metrics that would
be useful?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with Join

[William Bell]

Can a committer take a look at this please?


On Wed, Dec 30, 2015 at 4:35 PM, William Bell <billnb...@gmail.com> wrote:

> Yeah I ran it. It appears to only have an issue if the field for the core
> on the outside is MultiValue, and the inside field is single value, and the
> order of the field has the code not as the first entry. Like:
>
>  practicing_specialties_codes:
> >  [
> > - "PS1010",
> > - "PS282",
> > - "*PS1226*"
> > ]
> >  }
>
> Here is the output of the debug:
>
>
>- join:
>{
>   - {!join from=prac_spec_code to=practicing_specialties_codes
>   fromIndex=autosuggest}*:*:
>   {
>  - time: 24,
>  - fromSetSize: 170179,
>  - toSetSize: 3194542,
>  - fromTermCount: 1091,
>  - fromTermTotalDf: 1091,
>  - fromTermDirectCount: 1091,
>  - fromTermHits: 1091,
>  - fromTermHitsTotalDf: 1091,
>  - toTermHits: 1069,
>  - toTermHitsTotalDf: 4002538,
>  - toTermDirectCount: 697,
>  - smallSetsDeferred: 1,
>  - toSetDocsAdded: 4002538
>  }
>   },
>
>
> On Wed, Dec 30, 2015 at 2:02 AM, Mikhail Khludnev <
> mkhlud...@griddynamics.com> wrote:
>
>> Hello,
>>
>> Have you considered to pass NOT {!join from=prac_spec_code
>> to=practicing_specialties_codes
>>  fromIndex=autosuggest}*:* as a q and look at debug output.
>> Also there was a trick for dealing with pure negative queries q=*:*
>> -foo:bar though it might be obsolete.
>>
>> On Tue, Dec 29, 2015 at 7:02 AM, William Bell <billnb...@gmail.com>
>> wrote:
>>
>> > I have having issues with {!join}. If the core have multiValued field
>> and
>> > the inner join does not have a multiValued field it does not find the
>> > ones...
>> >
>> > Solr 5.3.1... 5.3.1
>> >
>> > Example.
>> >
>> > PS1226 is in practicing_specialties_codes in providersearch core. This
>> > field is multiValued.
>> >
>> > in the autosuggest core there is NOT a field for PS1226 in there. This
>> > field is called prac_spec_code and is single values.
>> >
>> >
>> >
>> >
>> http://localhost:8983/solr/providersearch/select?q=*%3A*=json=true=practicing_specialties_codes:PS1226=practicing_specialties_codes
>> >
>> > I get:
>> >
>> >
>> >- docs:
>> >[
>> >   -
>> >   {
>> >  - practicing_specialties_codes:
>> >  [
>> > - "PS1010",
>> > - "PS282",
>> > - "PS1226"
>> > ]
>> >  }
>> >   ]
>> >
>> >
>> >
>> > In autosuggest there is nothing:
>> >
>> >
>> >
>> http://localhost:8983/solr/autosuggest/select?q=*%3A*=json=true=prac_spec_code:PS1226=prac_spec_code
>> >
>> > Nothing.
>> >
>> > Then a join should find what is in providersearch but missing in
>> > autosuggest.
>> >
>> >
>> >
>> http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
>> <http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20%7B!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest%7Dauto_type:PRACSPEC>
>> >
>> > or
>> >
>> >
>> >
>> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
>> <http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20%7B!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest%7Dauto_type:PRACSPEC>
>> >
>> > or
>> >
>> >
>> >
>> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}*:*
>> <http://hgsolr2sl1:8983

Re: Issue with if() statement

[William Bell]

Thanks that indeed works tt=${state1}. But that was just a passing question.

I see an empty "*case*", "default" and a "case.VAL". I think I have
anything I need. After rereading. We might want to add to "case.VAL"
something like "case.'state:CO'" with a single quote. Right now it does not
appear to handle spaces or : I digress.

I got it to work with {!switch}.

http://localhost:8983/solr/providersearchfull/select?wt=xml=state:%22NY%22=score=*:*={!switch%20case=$state2%20default=$state1%20v=$state1}

If the user passes state1, it will not be empty and the default will pick
up all 50 states.

If the user does not pass state1, but passes state2, the case empty will
pick up since $state1 is empty, and it works.

http://localhost:8983/solr/providersearchfull/select?wt=xml=state:%22CO%22=score=*:*={!switch%20case=$state2%20default=$state1%20v=$state1}

The only that that changes is if I set as a param state1 or state2.

If I send both state1 and state2, state1 takes precedence.

http://localhost:8983/solr/providersearchfull/select?wt=xml=state:%22NY%22=state:%22CO%22=score=*:*={!switch%20case=$state2%20default=$state1%20v=$state1}

Kinda interesting.

The query I sent earlier : {!orquery}query1,query2 will return query1 is it
is not empty, otherwise if query1 is empty it will run query2. That was the
code I sent last night.

I am going to send some more code, since I could not get TermQuery to work
like I have it in PayLoad. And need some help debugging that.

Thanks!



On Sun, Jan 3, 2016 at 11:32 AM, Erik Hatcher <erik.hatc...@gmail.com>
wrote:

> I haven’t fully digested this thread, but wanted to comment on this one.
> =$something doesn’t substitute.  Only “local” params (inside local
> param syntax curly brackets) do that.  If you want to substitute a raw
> parameter use curly brackets.  You could, in this example, use
> =${state1}, I believe, and it would substitute.  Confusingly, local
> param curly brackets are _not_ the same as param substitution curly
> brackets.
>
> I am skeptical that anything custom is needed for what you’re trying to
> do, but again I haven’t fully digested what’s going on here yet.  There is
> some param substitution and {!switch} voodoo in example/files that ships
> with Solr, and that might be helpful.  Check out how that works.  I’m in
> the polishing stages of a a collaborative blog post on example/files that
> will detail these tricks a bit - will publish that in the next few days.
>
> —
> Erik Hatcher, Senior Solutions Architect
> http://www.lucidworks.com <http://www.lucidworks.com/>
>
>
>
> > On Jan 1, 2016, at 4:15 PM, William Bell <billnb...@gmail.com> wrote:
> >
> > Another weirdness:
> >
> >
> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
> >
> > That does not return anything.
> >
> > But if I set v=$state1 I get results.
> >
> > Can I not set equivalent variables?
> >
> >
> >
> > On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com>
> wrote:
> >
> >> Example.
> >>
> >>
> >>
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
> >> <
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D
> >
> >>
> >> This return 236,000
> >>
> >>
> >>
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
> >> <
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D
> >
> >>
> >> This returns 10,000
> >>
> >> I want to put an IF statement around which v to use.
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com>
> wrote:
> >>
> >>> Sure.
> >>>
> >>> If the state:NY returns results filter by state:NY, if it does not,
> then
> >>> use state:CO. If we have results in NY, use it, otherwise use CO.
> >>>
> >>> OK?
> >>>
> >>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
> >>>
> >>>>
> >>>>
> >>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
> >>>>> We are getting weird results with if(exists(a),b,c). We are getting
> >>>> b+c!!
> >>>>>
> >>>>

Re: Add support in FacetsComponent for facet.method=uif

[William Bell]

Interesting that facet.method=dv or facet.method=uif. What is the
difference?

On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson <jej2...@gmail.com> wrote:

> For those interested I created a separate jira issue for this but forgot to
> attach earlier.
>
> https://issues.apache.org/jira/browse/SOLR-8466
> On Jan 2, 2016 8:45 PM, "William Bell" <billnb...@gmail.com> wrote:
>
> > Yes we would like backward compatibility. We cannot switch all the facet
> > fields to DocValues and our faceting is slow.
> >
> > Please...
> >
> > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson <jej2...@gmail.com> wrote:
> >
> > > Is there any interest in this?  While i think it's important and inline
> > > with faceting available in the new json facet api, I've seen no
> > discussion
> > > on it so I'm wondering if it's best I add support for this using a
> custom
> > > facet component even though the majority of the component will be a
> copy
> > > which is prefer to not need to maintain separately.
> > >
> > > Jamie
> > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" <jej2...@gmail.com> wrote:
> > >
> > > > I had previously piggybacked on another post, but I think it may have
> > > been
> > > > lost there.  I had a need to do UnInvertedField based faceting in the
> > > > FacetsComponent and as such started looking at what would be required
> > to
> > > > implement something similar to what the JSON Facets based API does in
> > > this
> > > > regard.  The patch that I have in this regard works and is attached
> to
> > > > https://issues.apache.org/jira/browse/SOLR-8096, is that appropriate
> > or
> > > > should I create a new ticket to specifically add this support?
> > > >
> > > > -Jamie
> > > >
> > >
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Add support in FacetsComponent for facet.method=uif

[William Bell]

Ok the path appears to have dv and uif in there.?

On Sun, Jan 3, 2016 at 4:40 PM, Jamie Johnson <jej2...@gmail.com> wrote:

> The patch adds facet.method=uif and then delegates all of the work to the
> JSON Faceting API to do the work.  I had originally added a facet.method=dv
> and made the original facet.method=fc work using the UnInvertedField but
> wanted to avoid making a change that would introduce unexpected behavior.
> While I think it's strange that facet.method=dv does not exist and fc
> defaults to dv I think if we wanted to change that it should be done in
> another ticket.
>
> On Sun, Jan 3, 2016 at 4:18 PM, William Bell <billnb...@gmail.com> wrote:
>
> > Interesting that facet.method=dv or facet.method=uif. What is the
> > difference?
> >
> > On Sun, Jan 3, 2016 at 6:44 AM, Jamie Johnson <jej2...@gmail.com> wrote:
> >
> > > For those interested I created a separate jira issue for this but
> forgot
> > to
> > > attach earlier.
> > >
> > > https://issues.apache.org/jira/browse/SOLR-8466
> > > On Jan 2, 2016 8:45 PM, "William Bell" <billnb...@gmail.com> wrote:
> > >
> > > > Yes we would like backward compatibility. We cannot switch all the
> > facet
> > > > fields to DocValues and our faceting is slow.
> > > >
> > > > Please...
> > > >
> > > > On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson <jej2...@gmail.com>
> > wrote:
> > > >
> > > > > Is there any interest in this?  While i think it's important and
> > inline
> > > > > with faceting available in the new json facet api, I've seen no
> > > > discussion
> > > > > on it so I'm wondering if it's best I add support for this using a
> > > custom
> > > > > facet component even though the majority of the component will be a
> > > copy
> > > > > which is prefer to not need to maintain separately.
> > > > >
> > > > > Jamie
> > > > > On Dec 22, 2015 12:37 PM, "Jamie Johnson" <jej2...@gmail.com>
> wrote:
> > > > >
> > > > > > I had previously piggybacked on another post, but I think it may
> > have
> > > > > been
> > > > > > lost there.  I had a need to do UnInvertedField based faceting in
> > the
> > > > > > FacetsComponent and as such started looking at what would be
> > required
> > > > to
> > > > > > implement something similar to what the JSON Facets based API
> does
> > in
> > > > > this
> > > > > > regard.  The patch that I have in this regard works and is
> attached
> > > to
> > > > > > https://issues.apache.org/jira/browse/SOLR-8096, is that
> > appropriate
> > > > or
> > > > > > should I create a new ticket to specifically add this support?
> > > > > >
> > > > > > -Jamie
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Bill Bell
> > > > billnb...@gmail.com
> > > > cell 720-256-8076
> > > >
> > >
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

OK. I was able to get it to work by looking at {!switch} code. I had a heck
of a time trying to get parse() to work as a separate class.

q.add(new TermQuery(new Term("state", "CO"), Occur.MUST);
return q;

That never did anything.

Here is some code that worked...

http://localhost:8983/solr/providersearchfull/select?wt=json=state:NY=ss,score=state={!orquery}$state,$state1

Notice $state was not sent... It works.

Thoughts?

package hg.parsers;

import org.apache.commons.lang.StringUtils;
import org.apache.lucene.index.Term;
import org.apache.lucene.search.BooleanClause.Occur;
import org.apache.lucene.search.BooleanQuery;
import org.apache.solr.common.params.CommonParams;
import org.apache.lucene.search.TermQuery;
import org.apache.lucene.search.BooleanClause;
import org.apache.solr.parser.QueryParser;
import org.apache.lucene.search.Query;
import org.apache.lucene.search.spans.SpanTermQuery;
import org.apache.solr.common.params.SolrParams;
import org.apache.solr.search.QueryParsing;
import org.apache.solr.common.util.NamedList;
import org.apache.solr.request.SolrQueryRequest;
import org.apache.solr.search.QParser;
import org.apache.solr.search.QParserPlugin;
import org.apache.solr.search.SyntaxError;

/**
 * Created by bbell on 1/3/16.
 */
public class OrQueryParserPlugin extends QParserPlugin {
//public static String NAME = "lucene";

@Override
public void init(NamedList args) {
}

@Override
public QParser createParser(String qstr, SolrParams localParams,
SolrParams params, SolrQueryRequest req) {
return new QParser(qstr, localParams, params, req) {
QParser subParser;

@Override
public Query parse() throws SyntaxError {
String v = localParams.get(QueryParsing.V);
if (v == null || v.length() == 0) return null;
String subQ = null;

String[] arr = v.split(",");
System.out.println("parse arr:" + arr.length + "," + v);
if (arr != null && arr.length == 2) {
System.out.println("parse arr[0].length:" +
arr[0].length());
System.out.println("parse arr[1].length:" +
arr[1].length());
if (arr[0].trim().length() > 0) {
System.out.println("parse arr[0] params:" +
params.get(arr[0].trim()));
if (arr[0].trim().substring(0, 1).equals("$")) {
String val = params.get(arr[0].trim().substring(1));
if (val != null && val.length() > 0) {
subQ = val;
System.out.println("parse arr[0]:" +
arr[0].trim() + "," + subQ);
}
} else {
subQ = arr[0].trim();
System.out.println("parse arr[0]:" + subQ);
}
}
if (subQ == null && arr[1].trim().length() > 0) {
System.out.println("parse arr[1] params:" +
params.get(arr[1].trim()));
if (arr[1].trim().substring(0, 1).equals("$")) {
String val = params.get(arr[1].trim().substring(1));
if (val != null && val.length() > 0) {
subQ = val;
System.out.println("parse arr[1]:" +
arr[1].trim() + "," + subQ);
}
} else {
subQ = arr[1].trim();
System.out.println("parse arr[1]:" + subQ);
}
}
} else {
subQ = null;
}

subParser = subQuery(subQ, null);
return subParser.getQuery();
}
@Override
public String[] getDefaultHighlightFields() {
return subParser.getDefaultHighlightFields();
}

@Override
public Query getHighlightQuery() throws SyntaxError {
return subParser.getHighlightQuery();
}

@Override
public void addDebugInfo(NamedList debugInfo) {
subParser.addDebugInfo(debugInfo);
}
};
}
}



.

On Sat, Jan 2, 2016 at 5:03 AM, Upayavira <u...@odoko.co.uk> wrote:

> Hrmph. I've got an Ant based codebase for building custom components
> against Solr. I've been asked on numerous occasions to publish this
> codebase. It is now at:
>
> https://github.com/upayavira/custom-solr-components
>
> There's no sample code in there yet. I'll see if 

Re: Add support in FacetsComponent for facet.method=uif

[William Bell]

Yes we would like backward compatibility. We cannot switch all the facet
fields to DocValues and our faceting is slow.

Please...

On Fri, Jan 1, 2016 at 7:41 AM, Jamie Johnson  wrote:

> Is there any interest in this?  While i think it's important and inline
> with faceting available in the new json facet api, I've seen no discussion
> on it so I'm wondering if it's best I add support for this using a custom
> facet component even though the majority of the component will be a copy
> which is prefer to not need to maintain separately.
>
> Jamie
> On Dec 22, 2015 12:37 PM, "Jamie Johnson"  wrote:
>
> > I had previously piggybacked on another post, but I think it may have
> been
> > lost there.  I had a need to do UnInvertedField based faceting in the
> > FacetsComponent and as such started looking at what would be required to
> > implement something similar to what the JSON Facets based API does in
> this
> > regard.  The patch that I have in this regard works and is attached to
> > https://issues.apache.org/jira/browse/SOLR-8096, is that appropriate or
> > should I create a new ticket to specifically add this support?
> >
> > -Jamie
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

Another weirdness:

http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}

That does not return anything.

But if I set v=$state1 I get results.

Can I not set equivalent variables?



On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com> wrote:

> Example.
>
>
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D>
>
> This return 236,000
>
>
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D>
>
> This returns 10,000
>
> I want to put an IF statement around which v to use.
>
>
>
>
>
>
> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com> wrote:
>
>> Sure.
>>
>> If the state:NY returns results filter by state:NY, if it does not, then
>> use state:CO. If we have results in NY, use it, otherwise use CO.
>>
>> OK?
>>
>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>>
>>>
>>>
>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>>> > We are getting weird results with if(exists(a),b,c). We are getting
>>> b+c!!
>>> >
>>> >
>>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>>> <http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),%7B!lucene%20v=$state1%7D,%7B!lucene%20v=$state%7D)>
>>> >
>>> > I am getting NY and CO!
>>> >
>>> > I only want $state1, which is NY.
>>> >
>>> > Any other ways to craft this?
>>>
>>> Does this work at all?
>>>
>>> The if() function is a function query that can be used to sort, boost
>>> and as calculated fields. I haven't seen them used in filtering.
>>>
>>> Also, the query() function does *not* do a query, it just says "what
>>> would this document score for this query?"
>>>
>>> Can you describe in English what you are trying to do?
>>>
>>> Upayavira
>>>
>>
>>
>>
>> --
>> Bill Bell
>> billnb...@gmail.com
>> cell 720-256-8076
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

This all started with me trying to use {!switch} to indicate with CO or NY
to use. if we pass state1, that is supposed to take precedence, but if
nothing is returned, then use state Make sense now?

I could not find a way to check for strlen($state1). Which is what I want
in the case statement. This just does not work...
Something like the following is what I am trying to do:


fq={!switch case."anything"=$state1
default=$state
 v=$state1}

So I tried the following to set all 50 states:


fq={!switch case.all='*:*'
 case.CO='state:CO'
 case.NY='state:NY'
 v=$state1}


Obviously Oregon (OR!) is an issue, and it won't work at all plus



On Fri, Jan 1, 2016 at 2:15 PM, William Bell <billnb...@gmail.com> wrote:

> Another weirdness:
>
>
> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
> <http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1=%7B!lucene%20v=$tt%7D>
>
> That does not return anything.
>
> But if I set v=$state1 I get results.
>
> Can I not set equivalent variables?
>
>
>
> On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com> wrote:
>
>> Example.
>>
>>
>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D>
>>
>> This return 236,000
>>
>>
>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D>
>>
>> This returns 10,000
>>
>> I want to put an IF statement around which v to use.
>>
>>
>>
>>
>>
>>
>> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com> wrote:
>>
>>> Sure.
>>>
>>> If the state:NY returns results filter by state:NY, if it does not, then
>>> use state:CO. If we have results in NY, use it, otherwise use CO.
>>>
>>> OK?
>>>
>>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>
>>>>
>>>>
>>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>>>> > We are getting weird results with if(exists(a),b,c). We are getting
>>>> b+c!!
>>>> >
>>>> >
>>>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>>>> <http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),%7B!lucene%20v=$state1%7D,%7B!lucene%20v=$state%7D)>
>>>> >
>>>> > I am getting NY and CO!
>>>> >
>>>> > I only want $state1, which is NY.
>>>> >
>>>> > Any other ways to craft this?
>>>>
>>>> Does this work at all?
>>>>
>>>> The if() function is a function query that can be used to sort, boost
>>>> and as calculated fields. I haven't seen them used in filtering.
>>>>
>>>> Also, the query() function does *not* do a query, it just says "what
>>>> would this document score for this query?"
>>>>
>>>> Can you describe in English what you are trying to do?
>>>>
>>>> Upayavira
>>>>
>>>
>>>
>>>
>>> --
>>> Bill Bell
>>> billnb...@gmail.com
>>> cell 720-256-8076
>>>
>>
>>
>>
>> --
>> Bill Bell
>> billnb...@gmail.com
>> cell 720-256-8076
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

Also ne: not equal.

This is like KSH.

On Fri, Jan 1, 2016 at 2:40 PM, William Bell <billnb...@gmail.com> wrote:

> Forgot last bit:
>
> fq={!switch case.gt.0=$state1
>  default=$state
>  func=len($state1)}
>
>
> On Fri, Jan 1, 2016 at 2:39 PM, William Bell <billnb...@gmail.com> wrote:
>
>> Maybe we can add function value on a switch?
>>
>> fq={!switch case.gt.0
>>  default=$state
>>  func=len($state1)}
>>
>>
>>
>> We could add: gt, lt, eq, ge, le ?
>>
>> gt: greater than
>> lt: less than
>> eq: equal
>> ge: greater than or equal
>> le: less than or equal
>>
>> ??
>>
>>
>>
>> On Fri, Jan 1, 2016 at 2:25 PM, William Bell <billnb...@gmail.com> wrote:
>>
>>> This all started with me trying to use {!switch} to indicate with CO or
>>> NY to use. if we pass state1, that is supposed to take precedence, but if
>>> nothing is returned, then use state Make sense now?
>>>
>>> I could not find a way to check for strlen($state1). Which is what I
>>> want in the case statement. This just does not work...
>>> Something like the following is what I am trying to do:
>>>
>>>
>>> fq={!switch case."anything"=$state1
>>> default=$state
>>>  v=$state1}
>>>
>>> So I tried the following to set all 50 states:
>>>
>>>
>>> fq={!switch case.all='*:*'
>>>  case.CO='state:CO'
>>>  case.NY='state:NY'
>>>  v=$state1}
>>>
>>>
>>> Obviously Oregon (OR!) is an issue, and it won't work at all plus
>>>
>>>
>>>
>>> On Fri, Jan 1, 2016 at 2:15 PM, William Bell <billnb...@gmail.com>
>>> wrote:
>>>
>>>> Another weirdness:
>>>>
>>>>
>>>> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
>>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1=%7B!lucene%20v=$tt%7D>
>>>>
>>>> That does not return anything.
>>>>
>>>> But if I set v=$state1 I get results.
>>>>
>>>> Can I not set equivalent variables?
>>>>
>>>>
>>>>
>>>> On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com>
>>>> wrote:
>>>>
>>>>> Example.
>>>>>
>>>>>
>>>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
>>>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D>
>>>>>
>>>>> This return 236,000
>>>>>
>>>>>
>>>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
>>>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D>
>>>>>
>>>>> This returns 10,000
>>>>>
>>>>> I want to put an IF statement around which v to use.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Sure.
>>>>>>
>>>>>> If the state:NY returns results filter by state:NY, if it does not,
>>>>>> then use state:CO. If we have results in NY, use it, otherwise use CO.
>>>>>>
>>>>>> OK?
>>>>>>
>>>>>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>>>>>>> > We are getting weird results with if(exists(a),b,c). We are
>>>>>>> getting b+c!!
>>>>>>> >
>>>>>>> >
>>>>>>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state

Re: Issue with if() statement

[William Bell]

Example.

http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}

This return 236,000

http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}

This returns 10,000

I want to put an IF statement around which v to use.






On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com> wrote:

> Sure.
>
> If the state:NY returns results filter by state:NY, if it does not, then
> use state:CO. If we have results in NY, use it, otherwise use CO.
>
> OK?
>
> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>
>>
>>
>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>> > We are getting weird results with if(exists(a),b,c). We are getting
>> b+c!!
>> >
>> >
>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>> <http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),%7B!lucene%20v=$state1%7D,%7B!lucene%20v=$state%7D)>
>> >
>> > I am getting NY and CO!
>> >
>> > I only want $state1, which is NY.
>> >
>> > Any other ways to craft this?
>>
>> Does this work at all?
>>
>> The if() function is a function query that can be used to sort, boost
>> and as calculated fields. I haven't seen them used in filtering.
>>
>> Also, the query() function does *not* do a query, it just says "what
>> would this document score for this query?"
>>
>> Can you describe in English what you are trying to do?
>>
>> Upayavira
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

Maybe we can add function value on a switch?

fq={!switch case.gt.0
 default=$state
 func=len($state1)}



We could add: gt, lt, eq, ge, le ?

gt: greater than
lt: less than
eq: equal
ge: greater than or equal
le: less than or equal

??



On Fri, Jan 1, 2016 at 2:25 PM, William Bell <billnb...@gmail.com> wrote:

> This all started with me trying to use {!switch} to indicate with CO or NY
> to use. if we pass state1, that is supposed to take precedence, but if
> nothing is returned, then use state Make sense now?
>
> I could not find a way to check for strlen($state1). Which is what I want
> in the case statement. This just does not work...
> Something like the following is what I am trying to do:
>
>
> fq={!switch case."anything"=$state1
> default=$state
>  v=$state1}
>
> So I tried the following to set all 50 states:
>
>
> fq={!switch case.all='*:*'
>  case.CO='state:CO'
>  case.NY='state:NY'
>  v=$state1}
>
>
> Obviously Oregon (OR!) is an issue, and it won't work at all plus
>
>
>
> On Fri, Jan 1, 2016 at 2:15 PM, William Bell <billnb...@gmail.com> wrote:
>
>> Another weirdness:
>>
>>
>> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
>> <http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1=%7B!lucene%20v=$tt%7D>
>>
>> That does not return anything.
>>
>> But if I set v=$state1 I get results.
>>
>> Can I not set equivalent variables?
>>
>>
>>
>> On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com> wrote:
>>
>>> Example.
>>>
>>>
>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D>
>>>
>>> This return 236,000
>>>
>>>
>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D>
>>>
>>> This returns 10,000
>>>
>>> I want to put an IF statement around which v to use.
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com>
>>> wrote:
>>>
>>>> Sure.
>>>>
>>>> If the state:NY returns results filter by state:NY, if it does not,
>>>> then use state:CO. If we have results in NY, use it, otherwise use CO.
>>>>
>>>> OK?
>>>>
>>>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>>>>> > We are getting weird results with if(exists(a),b,c). We are getting
>>>>> b+c!!
>>>>> >
>>>>> >
>>>>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>>>>> <http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),%7B!lucene%20v=$state1%7D,%7B!lucene%20v=$state%7D)>
>>>>> >
>>>>> > I am getting NY and CO!
>>>>> >
>>>>> > I only want $state1, which is NY.
>>>>> >
>>>>> > Any other ways to craft this?
>>>>>
>>>>> Does this work at all?
>>>>>
>>>>> The if() function is a function query that can be used to sort, boost
>>>>> and as calculated fields. I haven't seen them used in filtering.
>>>>>
>>>>> Also, the query() function does *not* do a query, it just says "what
>>>>> would this document score for this query?"
>>>>>
>>>>> Can you describe in English what you are trying to do?
>>>>>
>>>>> Upayavira
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Bill Bell
>>>> billnb...@gmail.com
>>>> cell 720-256-8076
>>>>
>>>
>>>
>>>
>>> --
>>> Bill Bell
>>> billnb...@gmail.com
>>> cell 720-256-8076
>>>
>>
>>
>>
>> --
>> Bill Bell
>> billnb...@gmail.com
>> cell 720-256-8076
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

Forgot last bit:

fq={!switch case.gt.0=$state1
 default=$state
 func=len($state1)}


On Fri, Jan 1, 2016 at 2:39 PM, William Bell <billnb...@gmail.com> wrote:

> Maybe we can add function value on a switch?
>
> fq={!switch case.gt.0
>  default=$state
>  func=len($state1)}
>
>
>
> We could add: gt, lt, eq, ge, le ?
>
> gt: greater than
> lt: less than
> eq: equal
> ge: greater than or equal
> le: less than or equal
>
> ??
>
>
>
> On Fri, Jan 1, 2016 at 2:25 PM, William Bell <billnb...@gmail.com> wrote:
>
>> This all started with me trying to use {!switch} to indicate with CO or
>> NY to use. if we pass state1, that is supposed to take precedence, but if
>> nothing is returned, then use state Make sense now?
>>
>> I could not find a way to check for strlen($state1). Which is what I want
>> in the case statement. This just does not work...
>> Something like the following is what I am trying to do:
>>
>>
>> fq={!switch case."anything"=$state1
>> default=$state
>>  v=$state1}
>>
>> So I tried the following to set all 50 states:
>>
>>
>> fq={!switch case.all='*:*'
>>  case.CO='state:CO'
>>      case.NY='state:NY'
>>  v=$state1}
>>
>>
>> Obviously Oregon (OR!) is an issue, and it won't work at all plus
>>
>>
>>
>> On Fri, Jan 1, 2016 at 2:15 PM, William Bell <billnb...@gmail.com> wrote:
>>
>>> Another weirdness:
>>>
>>>
>>> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1=%7B!lucene%20v=$tt%7D>
>>>
>>> That does not return anything.
>>>
>>> But if I set v=$state1 I get results.
>>>
>>> Can I not set equivalent variables?
>>>
>>>
>>>
>>> On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com>
>>> wrote:
>>>
>>>> Example.
>>>>
>>>>
>>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
>>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D>
>>>>
>>>> This return 236,000
>>>>
>>>>
>>>> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state}
>>>> <http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state%7D>
>>>>
>>>> This returns 10,000
>>>>
>>>> I want to put an IF statement around which v to use.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Fri, Jan 1, 2016 at 1:52 PM, William Bell <billnb...@gmail.com>
>>>> wrote:
>>>>
>>>>> Sure.
>>>>>
>>>>> If the state:NY returns results filter by state:NY, if it does not,
>>>>> then use state:CO. If we have results in NY, use it, otherwise use CO.
>>>>>
>>>>> OK?
>>>>>
>>>>> On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:
>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
>>>>>> > We are getting weird results with if(exists(a),b,c). We are getting
>>>>>> b+c!!
>>>>>> >
>>>>>> >
>>>>>> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
>>>>>> <http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),%7B!lucene%20v=$state1%7D,%7B!lucene%20v=$state%7D)>
>>>>>> >
>>>>>> > I am getting NY and CO!
>>>>>> >
>>>>>> > I only want $state1, which is NY.
>>>>>> >
>>>>>> > Any other ways to craft this?
>>>>>>
>>>>>> Does this work at all?
>>>>>>
>>>>>> The if() function is a function query that can be used to sort, boost
>>>>>> and as calculated fields. I haven't seen them used in filtering.
>>>>>>
>>>>>> Also, the query() function does *not* do a query, it just says "what
>>>>>> would this document score for this query?"
>>>>>>
>>>>>> Can you describe in English what you are trying to do?
>>>>>>
>>>>>> Upayavira
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Bill Bell
>>>>> billnb...@gmail.com
>>>>> cell 720-256-8076
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Bill Bell
>>>> billnb...@gmail.com
>>>> cell 720-256-8076
>>>>
>>>
>>>
>>>
>>> --
>>> Bill Bell
>>> billnb...@gmail.com
>>> cell 720-256-8076
>>>
>>
>>
>>
>> --
>> Bill Bell
>> billnb...@gmail.com
>> cell 720-256-8076
>>
>
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with if() statement

[William Bell]

Sample code for a simple query parser?

On Fri, Jan 1, 2016 at 3:08 PM, Upayavira <u...@odoko.co.uk> wrote:

> You've got to think of what the queries will be that you produce under
> the bonnet - what you are talking about is either a (!func} or a
> {!frange} query - both of which are inefficient as they must scan the
> entire resultset.
>
> However, what you're trying to do is evaluate the length of the *input
> value*, not a value in the documents being searched, so this doesn't
> really apply - you're talking about doing some work before your query
> happens.
>
> Again, I'm not sure of a way to do this on a stock Solr, but with a
> little coding it should be achievable - you could extend the switch
> query parser, or you could write your own - they're not that hard to
> create. Or, you could create a search component that operates before the
> QueryComponent does.
>
> A simple queryparser of your own would seem like a pretty lightweight
> thing - probably 20 lines of code or less.
>
> Upayavira
>
> On Fri, Jan 1, 2016, at 09:40 PM, William Bell wrote:
> > Forgot last bit:
> >
> > fq={!switch case.gt.0=$state1
> >  default=$state
> >      func=len($state1)}
> >
> >
> > On Fri, Jan 1, 2016 at 2:39 PM, William Bell <billnb...@gmail.com>
> wrote:
> >
> > > Maybe we can add function value on a switch?
> > >
> > > fq={!switch case.gt.0
> > >  default=$state
> > >  func=len($state1)}
> > >
> > >
> > >
> > > We could add: gt, lt, eq, ge, le ?
> > >
> > > gt: greater than
> > > lt: less than
> > > eq: equal
> > > ge: greater than or equal
> > > le: less than or equal
> > >
> > > ??
> > >
> > >
> > >
> > > On Fri, Jan 1, 2016 at 2:25 PM, William Bell <billnb...@gmail.com>
> wrote:
> > >
> > >> This all started with me trying to use {!switch} to indicate with CO
> or
> > >> NY to use. if we pass state1, that is supposed to take precedence,
> but if
> > >> nothing is returned, then use state Make sense now?
> > >>
> > >> I could not find a way to check for strlen($state1). Which is what I
> want
> > >> in the case statement. This just does not work...
> > >> Something like the following is what I am trying to do:
> > >>
> > >>
> > >> fq={!switch case."anything"=$state1
> > >>         default=$state
> > >>  v=$state1}
> > >>
> > >> So I tried the following to set all 50 states:
> > >>
> > >>
> > >> fq={!switch case.all='*:*'
> > >>  case.CO='state:CO'
> > >>  case.NY='state:NY'
> > >>  v=$state1}
> > >>
> > >>
> > >> Obviously Oregon (OR!) is an issue, and it won't work at all plus
> > >>
> > >>
> > >>
> > >> On Fri, Jan 1, 2016 at 2:15 PM, William Bell <billnb...@gmail.com>
> wrote:
> > >>
> > >>> Another weirdness:
> > >>>
> > >>>
> > >>>
> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1={!lucene%20v=$tt}
> > >>> <
> http://localhost:8983/solr/providersearch/select?wt=json=state:CO=state:NY=*=*:*=$state1=%7B!lucene%20v=$tt%7D
> >
> > >>>
> > >>> That does not return anything.
> > >>>
> > >>> But if I set v=$state1 I get results.
> > >>>
> > >>> Can I not set equivalent variables?
> > >>>
> > >>>
> > >>>
> > >>> On Fri, Jan 1, 2016 at 2:07 PM, William Bell <billnb...@gmail.com>
> > >>> wrote:
> > >>>
> > >>>> Example.
> > >>>>
> > >>>>
> > >>>>
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*={!lucene%20v=$state1}
> > >>>> <
> http://localhost:8983/solr/providersearch/select?wt=json=state:%22CO%22=state:%22NY%22=ss,score=*:*=%7B!lucene%20v=$state1%7D
> >
> > >>>>
> > >>>> This return 236,000
> > >>>>
> > >>>>
> > >>>>
&

Re: Issue with if() statement

[William Bell]

Sure.

If the state:NY returns results filter by state:NY, if it does not, then
use state:CO. If we have results in NY, use it, otherwise use CO.

OK?

On Fri, Jan 1, 2016 at 1:15 PM, Upayavira <u...@odoko.co.uk> wrote:

>
>
> On Thu, Dec 31, 2015, at 11:50 PM, William Bell wrote:
> > We are getting weird results with if(exists(a),b,c). We are getting b+c!!
> >
> >
> http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})
> >
> > I am getting NY and CO!
> >
> > I only want $state1, which is NY.
> >
> > Any other ways to craft this?
>
> Does this work at all?
>
> The if() function is a function query that can be used to sort, boost
> and as calculated fields. I haven't seen them used in filtering.
>
> Also, the query() function does *not* do a query, it just says "what
> would this document score for this query?"
>
> Can you describe in English what you are trying to do?
>
> Upayavira
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Issue with if() statement

[William Bell]

We are getting weird results with if(exists(a),b,c). We are getting b+c!!

http://localhost:8983/solr/providersearch/select?q=*:*=json=state:%22CO%22=state:%22NY%22=if(exists(query($state1)),{!lucene%20v=$state1},{!lucene%20v=$state})

I am getting NY and CO!

I only want $state1, which is NY.

Any other ways to craft this?


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

SOLR 5.4.0?

[William Bell]

How is SOLR 5.4.0 ? I heard there was a quick 5.4.1 coming out?

Any major issues?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Issue with Join

[William Bell]

Yeah I ran it. It appears to only have an issue if the field for the core
on the outside is MultiValue, and the inside field is single value, and the
order of the field has the code not as the first entry. Like:

 practicing_specialties_codes:
>  [
> - "PS1010",
> - "PS282",
> - "*PS1226*"
> ]
>  }

Here is the output of the debug:


   - join:
   {
  - {!join from=prac_spec_code to=practicing_specialties_codes
  fromIndex=autosuggest}*:*:
  {
 - time: 24,
 - fromSetSize: 170179,
 - toSetSize: 3194542,
 - fromTermCount: 1091,
 - fromTermTotalDf: 1091,
 - fromTermDirectCount: 1091,
 - fromTermHits: 1091,
 - fromTermHitsTotalDf: 1091,
 - toTermHits: 1069,
 - toTermHitsTotalDf: 4002538,
 - toTermDirectCount: 697,
 - smallSetsDeferred: 1,
 - toSetDocsAdded: 4002538
 }
  },


On Wed, Dec 30, 2015 at 2:02 AM, Mikhail Khludnev <
mkhlud...@griddynamics.com> wrote:

> Hello,
>
> Have you considered to pass NOT {!join from=prac_spec_code
> to=practicing_specialties_codes
>  fromIndex=autosuggest}*:* as a q and look at debug output.
> Also there was a trick for dealing with pure negative queries q=*:*
> -foo:bar though it might be obsolete.
>
> On Tue, Dec 29, 2015 at 7:02 AM, William Bell <billnb...@gmail.com> wrote:
>
> > I have having issues with {!join}. If the core have multiValued field and
> > the inner join does not have a multiValued field it does not find the
> > ones...
> >
> > Solr 5.3.1... 5.3.1
> >
> > Example.
> >
> > PS1226 is in practicing_specialties_codes in providersearch core. This
> > field is multiValued.
> >
> > in the autosuggest core there is NOT a field for PS1226 in there. This
> > field is called prac_spec_code and is single values.
> >
> >
> >
> >
> http://localhost:8983/solr/providersearch/select?q=*%3A*=json=true=practicing_specialties_codes:PS1226=practicing_specialties_codes
> >
> > I get:
> >
> >
> >- docs:
> >[
> >   -
> >   {
> >  - practicing_specialties_codes:
> >  [
> > - "PS1010",
> > - "PS282",
> > - "PS1226"
> > ]
> >  }
> >   ]
> >
> >
> >
> > In autosuggest there is nothing:
> >
> >
> >
> http://localhost:8983/solr/autosuggest/select?q=*%3A*=json=true=prac_spec_code:PS1226=prac_spec_code
> >
> > Nothing.
> >
> > Then a join should find what is in providersearch but missing in
> > autosuggest.
> >
> >
> >
> http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
> >
> > or
> >
> >
> >
> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
> >
> > or
> >
> >
> >
> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}*:*
> >
> > I also tried *:* AND NOT {!join}
> >
> > I get 0 results. This seems to be a bug.
> >
> > {
> >
> >- responseHeader:
> >{
> >   - status: 0,
> >   - QTime: 178,
> >   - params:
> >   {
> >  - q: "*:*",
> >  - fl: "practicing_specialties_codes",
> >  - fq: "NOT {!join from=prac_spec_code
> >  to=practicing_specialties_codes fromIndex=autosuggest}*:*",
> >  - rows: "10",
> >  - wt: "json",
> >  - debugQuery: "true"
> >  }
> >   },
> >- response:
> >{
> >   - numFound: 0,
> >   - start: 0,
> >   - docs: [ ]
> >   },
> >- debug:
> >{
> >   - rawquerystring: "*:*",
> >   - querystring: "*:*",
> >   - parsedquery: "MatchAllDocsQuery(*:*)",
> >   - parsedquery_toString: "*:*",
> >   - explain: { },
> >   - QParser: "LuceneQParser

Re: Issue with Join

[William Bell]

Thoughts?

I can duplicate it at will...

On Mon, Dec 28, 2015 at 9:02 PM, William Bell <billnb...@gmail.com> wrote:

> I have having issues with {!join}. If the core have multiValued field and
> the inner join does not have a multiValued field it does not find the
> ones...
>
> Solr 5.3.1... 5.3.1
>
> Example.
>
> PS1226 is in practicing_specialties_codes in providersearch core. This
> field is multiValued.
>
> in the autosuggest core there is NOT a field for PS1226 in there. This
> field is called prac_spec_code and is single values.
>
>
>
> http://localhost:8983/solr/providersearch/select?q=*%3A*=json=true=practicing_specialties_codes:PS1226=practicing_specialties_codes
>
> I get:
>
>
>- docs:
>[
>   -
>   {
>  - practicing_specialties_codes:
>  [
> - "PS1010",
> - "PS282",
> - "PS1226"
> ]
>  }
>   ]
>
>
>
> In autosuggest there is nothing:
>
>
> http://localhost:8983/solr/autosuggest/select?q=*%3A*=json=true=prac_spec_code:PS1226=prac_spec_code
>
> Nothing.
>
> Then a join should find what is in providersearch but missing in
> autosuggest.
>
>
> http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
> <http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20%7B!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest%7Dauto_type:PRACSPEC>
>
> or
>
>
> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC
> <http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20%7B!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest%7Dauto_type:PRACSPEC>
>
> or
>
>
> http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}*:*
> <http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20%7B!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest%7D*:*>
>
> I also tried *:* AND NOT {!join}
>
> I get 0 results. This seems to be a bug.
>
> {
>
>- responseHeader:
>{
>   - status: 0,
>   - QTime: 178,
>   - params:
>   {
>  - q: "*:*",
>  - fl: "practicing_specialties_codes",
>  - fq: "NOT {!join from=prac_spec_code
>  to=practicing_specialties_codes fromIndex=autosuggest}*:*",
>  - rows: "10",
>  - wt: "json",
>  - debugQuery: "true"
>  }
>   },
>- response:
>{
>   - numFound: 0,
>   - start: 0,
>   - docs: [ ]
>   },
>- debug:
>{
>   - rawquerystring: "*:*",
>   - querystring: "*:*",
>   - parsedquery: "MatchAllDocsQuery(*:*)",
>   - parsedquery_toString: "*:*",
>   - explain: { },
>   - QParser: "LuceneQParser",
>   - filter_queries:
>   [
>  - "NOT {!join from=prac_spec_code
>  to=practicing_specialties_codes fromIndex=autosuggest}*:*"
>  ],
>   - parsed_filter_queries:
>   [
>  - "-JoinQuery({!join from=prac_spec_code
>  to=practicing_specialties_codes fromIndex=autosuggest}*:*)"
>  ],
>   - timing:
>   {
>  - time: 177,
>  - prepare:
>  {
> - time: 0,
> - query:
> {
>- time: 0
>},
> - facet:
> {
>- time: 0
>},
> - facet_module:
> {
>- time: 0
>},
> - mlt:
> {
>- time: 0
>},
> - highlight:
> {
>- time: 0
>},
> - stats:
> {
>- time: 0
>},
> - expand:
> 

Re: Memory Usage increases by a lot during and after optimization .

[William Bell]

Question: does anyone have example good merge settings for solrconfig ? To
keep the number of segments small like 6?

On Tue, Dec 29, 2015 at 8:49 PM, Yonik Seeley  wrote:

> Some people also want to control when major segment merges happen, and
> optimizing at a known time helps prevent a major merge at an unknown
> time (which can be equivalent to an optimize/forceMerge).
>
> The benefits of optimizing (and having fewer segments to search
> across) will vary depending on the requests.
> Normal full-text searches will see little benefit (merging a few terms
> across many segments is not expensive), while other operations that
> need to deal with many terms, like faceting, may see bigger speedups.
>
> -Yonik
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Issue with Join

[William Bell]

I have having issues with {!join}. If the core have multiValued field and
the inner join does not have a multiValued field it does not find the
ones...

Solr 5.3.1... 5.3.1

Example.

PS1226 is in practicing_specialties_codes in providersearch core. This
field is multiValued.

in the autosuggest core there is NOT a field for PS1226 in there. This
field is called prac_spec_code and is single values.


http://localhost:8983/solr/providersearch/select?q=*%3A*=json=true=practicing_specialties_codes:PS1226=practicing_specialties_codes

I get:


   - docs:
   [
  -
  {
 - practicing_specialties_codes:
 [
- "PS1010",
- "PS282",
- "PS1226"
]
 }
  ]



In autosuggest there is nothing:

http://localhost:8983/solr/autosuggest/select?q=*%3A*=json=true=prac_spec_code:PS1226=prac_spec_code

Nothing.

Then a join should find what is in providersearch but missing in
autosuggest.

http://localhost:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes:PS1226=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC

or

http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}auto_type:PRACSPEC

or

http://hgsolr2sl1:8983/solr/providersearch/select?debugQuery=true=json=*:*=10=practicing_specialties_codes=NOT%20{!join%20from=prac_spec_code%20to=practicing_specialties_codes%20fromIndex=autosuggest}*:*

I also tried *:* AND NOT {!join}

I get 0 results. This seems to be a bug.

{

   - responseHeader:
   {
  - status: 0,
  - QTime: 178,
  - params:
  {
 - q: "*:*",
 - fl: "practicing_specialties_codes",
 - fq: "NOT {!join from=prac_spec_code
 to=practicing_specialties_codes fromIndex=autosuggest}*:*",
 - rows: "10",
 - wt: "json",
 - debugQuery: "true"
 }
  },
   - response:
   {
  - numFound: 0,
  - start: 0,
  - docs: [ ]
  },
   - debug:
   {
  - rawquerystring: "*:*",
  - querystring: "*:*",
  - parsedquery: "MatchAllDocsQuery(*:*)",
  - parsedquery_toString: "*:*",
  - explain: { },
  - QParser: "LuceneQParser",
  - filter_queries:
  [
 - "NOT {!join from=prac_spec_code to=practicing_specialties_codes
 fromIndex=autosuggest}*:*"
 ],
  - parsed_filter_queries:
  [
 - "-JoinQuery({!join from=prac_spec_code
 to=practicing_specialties_codes fromIndex=autosuggest}*:*)"
 ],
  - timing:
  {
 - time: 177,
 - prepare:
 {
- time: 0,
- query:
{
   - time: 0
   },
- facet:
{
   - time: 0
   },
- facet_module:
{
   - time: 0
   },
- mlt:
{
   - time: 0
   },
- highlight:
{
   - time: 0
   },
- stats:
{
   - time: 0
   },
- expand:
{
   - time: 0
   },
- debug:
{
   - time: 0
   }
},
 - process:
 {
- time: 177,
- query:
{
   - time: 177
   },
- facet:
{
   - time: 0
   },
- facet_module:
{
   - time: 0
   },
- mlt:
{
   - time: 0
   },
- highlight:
{
   - time: 0
   },
- stats:
{
   - time: 0
   },
- expand:
{
   - time: 0
   },
- debug:
{
   - time: 0
   }
}
 }
  }

}




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: How to check when a search exceeds the threshold of timeAllowed parameter

[William Bell]

Great I must have missed that.

On Wed, Dec 23, 2015 at 9:41 AM, Jeff Wartes  wrote:

> Looks like it’ll set partialResults=true on your results if you hit the
> timeout.
>
> https://issues.apache.org/jira/browse/SOLR-502
>
> https://issues.apache.org/jira/browse/SOLR-5986
>
>
>
>
>
>
> On 12/22/15, 5:43 PM, "Vincenzo D'Amore"  wrote:
>
> >Well... I can write everything, but really all this just to understand
> >when timeAllowed
> >parameter trigger a partial answer? I mean, isn't there anything set in
> the
> >response when is partial?
> >
> >On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood 
> >wrote:
> >
> >> We need to know a LOT more about your site. Number of documents, size of
> >> index, frequency of updates, length of queries approximate size of
> server
> >> (CPUs, RAM, type of disk), version of Solr, version of Java, and
> features
> >> you are using (faceting, highlighting, etc.).
> >>
> >> After that, we’ll have more questions.
> >>
> >> wunder
> >> Walter Underwood
> >> wun...@wunderwood.org
> >> http://observer.wunderwood.org/  (my blog)
> >>
> >>
> >> > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore 
> >> wrote:
> >> >
> >> > Hi All,
> >> >
> >> > my website is under pressure, there is a big number of concurrent
> >> searches.
> >> > When the connected users are too many, the searches becomes so slow
> that
> >> in
> >> > some cases users have to wait many seconds.
> >> > The queue of searches becomes so long that, in same cases, servers are
> >> > blocked trying to serve all these requests.
> >> > As far as I know because some searches are very expensive, and when
> many
> >> > expensive searches clog the queue server becomes unresponsive.
> >> >
> >> > In order to quickly workaround this herd effect, I have added a
> >> > default timeAllowed to 15 seconds, and this seems help a lot.
> >> >
> >> > But during stress tests but I'm unable to understand when and what
> >> requests
> >> > are affected by timeAllowed parameter.
> >> >
> >> > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> >> > environment, given that partial results may be returned (if there are
> >> any),
> >> > how can I know when this happens? When the timeAllowed parameter
> trigger
> >> a
> >> > partial answer?
> >> >
> >> > Best regards,
> >> > Vincenzo
> >> >
> >> >
> >> >
> >> > --
> >> > Vincenzo D'Amore
> >> > email: v.dam...@gmail.com
> >> > skype: free.dev
> >> > mobile: +39 349 8513251
> >>
> >>
> >
> >
> >--
> >Vincenzo D'Amore
> >email: v.dam...@gmail.com
> >skype: free.dev
> >mobile: +39 349 8513251
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: How to check when a search exceeds the threshold of timeAllowed parameter

[William Bell]

I agree that when using timeAllowed in the header info there should be an
entry that indicates timeAllowed triggered.
This is the only reason why we have not used timeAllowed. So this is a
great suggestion. Something like: 1 ??
That would be great.


0
1
107

*:*
1000





On Tue, Dec 22, 2015 at 6:43 PM, Vincenzo D'Amore 
wrote:

> Well... I can write everything, but really all this just to understand
> when timeAllowed
> parameter trigger a partial answer? I mean, isn't there anything set in the
> response when is partial?
>
> On Wed, Dec 23, 2015 at 2:38 AM, Walter Underwood 
> wrote:
>
> > We need to know a LOT more about your site. Number of documents, size of
> > index, frequency of updates, length of queries approximate size of server
> > (CPUs, RAM, type of disk), version of Solr, version of Java, and features
> > you are using (faceting, highlighting, etc.).
> >
> > After that, we’ll have more questions.
> >
> > wunder
> > Walter Underwood
> > wun...@wunderwood.org
> > http://observer.wunderwood.org/  (my blog)
> >
> >
> > > On Dec 22, 2015, at 4:58 PM, Vincenzo D'Amore 
> > wrote:
> > >
> > > Hi All,
> > >
> > > my website is under pressure, there is a big number of concurrent
> > searches.
> > > When the connected users are too many, the searches becomes so slow
> that
> > in
> > > some cases users have to wait many seconds.
> > > The queue of searches becomes so long that, in same cases, servers are
> > > blocked trying to serve all these requests.
> > > As far as I know because some searches are very expensive, and when
> many
> > > expensive searches clog the queue server becomes unresponsive.
> > >
> > > In order to quickly workaround this herd effect, I have added a
> > > default timeAllowed to 15 seconds, and this seems help a lot.
> > >
> > > But during stress tests but I'm unable to understand when and what
> > requests
> > > are affected by timeAllowed parameter.
> > >
> > > Just be clear, I have configure timeAllowed parameter in a SolrCloud
> > > environment, given that partial results may be returned (if there are
> > any),
> > > how can I know when this happens? When the timeAllowed parameter
> trigger
> > a
> > > partial answer?
> > >
> > > Best regards,
> > > Vincenzo
> > >
> > >
> > >
> > > --
> > > Vincenzo D'Amore
> > > email: v.dam...@gmail.com
> > > skype: free.dev
> > > mobile: +39 349 8513251
> >
> >
>
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Best practices on monitoring Solr

[William Bell]

Sematext.com has a service for this...

Or just curl "http://localhost:8983/solr//select?q=*:*" to see
if it returns ?

On Tue, Dec 22, 2015 at 12:15 PM, Tiwari, Shailendra <
shailendra.tiw...@macmillan.com> wrote:

> Hi,
>
> Last week our Solr Search was un-responsive and we need to re-boot the
> server, but we were able to find out after customer complained about it.
> What's best way to monitor that search is working?
> We can always add Gomez alerts from UI.
> What are the best practices?
>
> Thanks
>
> Shail




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: API accessible without authentication even though Basic Auth Plugin is enabled

[William Bell]

Why would someone stay on 5.3.x instead of upgrading to 5.4? Why backport
when you can just upgrade?

On Tue, Dec 22, 2015 at 6:33 PM, Noble Paul  wrote:

> A 5.3.2 release is coming up which will back port the fixes introduced in
> 5.4
> On Dec 17, 2015 10:25 PM, "tine-2"  wrote:
>
> > Noble Paul നോബിള്‍  नोब्ळ् wrote
> > > It works as designed.
> > >
> > > Protect the read path [...]
> >
> > Works like described in 5.4.0, didn't work in 5.3.1, s.
> > https://issues.apache.org/jira/browse/SOLR-8408
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/API-accessible-without-authentication-even-though-Basic-Auth-Plugin-is-enabled-tp4244940p4246099.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

[William Bell]

Thanks Jamie.

On Sat, Dec 19, 2015 at 11:31 PM, Jamie Johnson <jej2...@gmail.com> wrote:

> Bill,
>
> Check out the patch attached to
> https://issues.apache.org/jira/browse/SOLR-8096.  I had considered making
> the method uif after I had done most of the work, it would be trivial to
> change and would probably be more aligned with not adding unexpected
> changes to people that are currently using fc.
>
> -Jamie
>
> On Sat, Dec 19, 2015 at 11:03 PM, William Bell <billnb...@gmail.com>
> wrote:
>
> > Can we add method=uif back when not using the JSON Facet API too?
> >
> > That would help a lot of people.
> >
> > On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley <ysee...@gmail.com> wrote:
> >
> > > On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore <v.dam...@gmail.com>
> > > wrote:
> > > > Hi all,
> > > >
> > > > given that solr 5.4 is finally released, is this what's more stable
> and
> > > > efficient version of solrcloud ?
> > > >
> > > > I have a website which receives many search requests. It serve
> normally
> > > > about 2000 concurrent requests, but sometime there are peak from 4000
> > to
> > > > 1 requests in few seconds.
> > > >
> > > > On January I'll have a chance to upgrade my old SolrCloud 4.8.1
> cluster
> > > to
> > > > a new brand version, but following this thread I read about the
> > problems
> > > > that can occur upgrading to latest version.
> > > >
> > > > I have seen that issue SOLR-7730 "speed-up faceting on doc values
> > fields"
> > > > is fixed in 5.4.
> > > >
> > > > I'm using standard faceting without docValues. Should I add docValues
> > in
> > > > order to benefit of such fix?
> > >
> > > You'll have to try it I think...
> > > DocValues have a lot of advantages (much less heap consumption, and
> > > much smaller overhead when opening a new searcher), but they can often
> > > be slower as well.
> > >
> > > Comparing 4x to 5x non-docvalues, top-level field caches were removed
> > > by lucene, and while that benefits certain things like NRT (opening a
> > > new searcher very often), it will hurt performance for other
> > > configurations.
> > >
> > > The JSON Facet API currently allows you to pick your strategy via the
> > > "method" param for multi-valued string fields without docvalues:
> > > "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> > > while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> > > "per-segment" strategy.
> > >
> > > -Yonik
> > >
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

[William Bell]

Can we add method=uif back when not using the JSON Facet API too?

That would help a lot of people.

On Thu, Dec 17, 2015 at 7:17 AM, Yonik Seeley  wrote:

> On Wed, Dec 16, 2015 at 4:57 AM, Vincenzo D'Amore 
> wrote:
> > Hi all,
> >
> > given that solr 5.4 is finally released, is this what's more stable and
> > efficient version of solrcloud ?
> >
> > I have a website which receives many search requests. It serve normally
> > about 2000 concurrent requests, but sometime there are peak from 4000 to
> > 1 requests in few seconds.
> >
> > On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster
> to
> > a new brand version, but following this thread I read about the problems
> > that can occur upgrading to latest version.
> >
> > I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> > is fixed in 5.4.
> >
> > I'm using standard faceting without docValues. Should I add docValues in
> > order to benefit of such fix?
>
> You'll have to try it I think...
> DocValues have a lot of advantages (much less heap consumption, and
> much smaller overhead when opening a new searcher), but they can often
> be slower as well.
>
> Comparing 4x to 5x non-docvalues, top-level field caches were removed
> by lucene, and while that benefits certain things like NRT (opening a
> new searcher very often), it will hurt performance for other
> configurations.
>
> The JSON Facet API currently allows you to pick your strategy via the
> "method" param for multi-valued string fields without docvalues:
> "uif" (UninvertedField) gets you the top-level strategy from Solr 4,
> while "dv" (DocValues built on-the-fly) gets you the NRT-friendly
> "per-segment" strategy.
>
> -Yonik
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: faceting is unusable slow since upgrade to 5.3.0

[William Bell]

Same question here

Wondering if faceting performance is fixed and how to take advantage of it ?

On Wed, Dec 16, 2015 at 2:57 AM, Vincenzo D'Amore 
wrote:

> Hi all,
>
> given that solr 5.4 is finally released, is this what's more stable and
> efficient version of solrcloud ?
>
> I have a website which receives many search requests. It serve normally
> about 2000 concurrent requests, but sometime there are peak from 4000 to
> 1 requests in few seconds.
>
> On January I'll have a chance to upgrade my old SolrCloud 4.8.1 cluster to
> a new brand version, but following this thread I read about the problems
> that can occur upgrading to latest version.
>
> I have seen that issue SOLR-7730 "speed-up faceting on doc values fields"
> is fixed in 5.4.
>
> I'm using standard faceting without docValues. Should I add docValues in
> order to benefit of such fix?
>
> Best regards,
> Vincenzo
>
>
>
> On Thu, Oct 8, 2015 at 2:22 PM, Mikhail Khludnev <
> mkhlud...@griddynamics.com
> > wrote:
>
> > Uwe, it's good to know! I mean that you've recovered. Take care!
> >
> > On Thu, Oct 8, 2015 at 1:24 PM, Uwe Reh 
> > wrote:
> >
> > > Sorry for the delay. I had an ugly flu.
> > >
> > > SOLR-7730 seems to work fine. Using docValues with Solr
> > > 5.4.0-2015-09-29_08-29-55 1705813 makes my faceted queries fast again.
> > > (90ms vs. 2ms) :-)
> > >
> > > Thanks
> > > Uwe
> > >
> > >
> > >
> > >
> > > Am 27.09.2015 um 20:32 schrieb Mikhail Khludnev:
> > >
> > >> On Sun, Sep 27, 2015 at 2:00 PM, Uwe Reh 
> > >> wrote:
> > >>
> > >> When 5.4 with SOLR-7730 will be released, I will start to use
> docValues.
> > >>> Going this way, seems more straight forward to me.
> > >>>
> > >>
> > >>
> > >> Sure. Giving your answers docValues facets has a really good chance to
> > >> perform in your index after SOLR-7730. It's really interesting to see
> > >> performance numbers on early 5.4 builds:
> > >>
> > >>
> >
> https://builds.apache.org/view/All/job/Solr-Artifacts-5.x/lastSuccessfulBuild/artifact/solr/package/
> > >>
> > >>
> > >
> >
> >
> > --
> > Sincerely yours
> > Mikhail Khludnev
> > Principal Engineer,
> > Grid Dynamics
> >
> > 
> > 
> >
>
>
>
> --
> Vincenzo D'Amore
> email: v.dam...@gmail.com
> skype: free.dev
> mobile: +39 349 8513251
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Migrating from cores to collections

[William Bell]

ok.

What about using DIH handler? Does it index in a SolrCloud setup ? Or how
would I convert a query to use SolrJ ?

On Mon, Nov 30, 2015 at 5:36 AM, Upayavira <u...@odoko.co.uk> wrote:

>
>
> On Sun, Nov 29, 2015, at 07:38 PM, William Bell wrote:
> > OK. Been using Cores for 4 years. Want to migrate to collections / Cloud.
> >
> > Do we have to change our queries?
> >
> > http://loadbalancer:8983/solr/corename/select?q=*:*
> >
> > What does this become once we have the collection sharded? Do we need a
> > Load Balancer or just point to one box and run the new query? Or would it
> > be better to hit the LB in case one machine is no longer good to go?
> >
> > http://loadbalancer:8983/solr/collectionname/select?q=*:*
> >
> > What features would not yet be ready for sharded setups with SolrCloud?
> > In
> > the past, facet counts were an issue, grouping? stats? as well as IDF for
> > sorting by scores. i.e. facet.field=specialties. We want the Cardiologist
> > specialty to have unique numbers across shards. So if shard1 has 4 people
> > with Cardiology, and shard2 has 2 people with Cardiology, we would want
> > the
> > number to be 6. We would want facet.sort to work on counts... I guess we
> > could index another collection for facets and just use 1 machine for
> > that?
> > But doesn't that defeat the purpose?
> >
> > What is the best walk thru for SOLR 5.3.1 ?
> >
> > Looking at https://wiki.apache.org/solr/SolrCloud
>
> 1. Your queries should stay (more or less) the same
> 2. If you name a collection the same as what you are using for a core,
> your base URL will remain the same
> 3. If you use SolrJ, then you would change to CloudSolrClient, which
> would feel quite different, but the SolrQuery objects should be
> interchangeable
> 4. If you use SolrJ, then you don't need a load balancer - SolrJ will do
> round robin against the Solr nodes for that collection. It will respond
> to failures far faster than an LB ever could (I've seen downed machines
> pulled in <200ms)
> 5. Regarding sharded setups, there's two scenarios to consider -
> distributed in general, and solrcloud in particular. Every search
> component must be enabled for distributed search (faceting,
> highlighting, grouping, etc, etc). Some of the newer ones may not have
> had distributed support implemented yet. Others, such as Joining, will
> require particular concern, and will work in only a subset of
> conditions.
> 6. For IDF, mostly, IDF balances itself across the shards. If it
> doesn't, then distributed IDF is available, but that has a cost in terms
> of additional network traffic.
> 7. Faceting should work just fine (as you describe) across shards. I
> would check specifically on newer faceting features though before
> assuming anything.
> 8. facet.sort+counts, have you tried it?
> 9. I would consider this to be a more up-to-date place to go:
> https://cwiki.apache.org/confluence/display/solr/SolrCloud
>
> Upayavira
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Migrating from cores to collections

[William Bell]

OK. Been using Cores for 4 years. Want to migrate to collections / Cloud.

Do we have to change our queries?

http://loadbalancer:8983/solr/corename/select?q=*:*

What does this become once we have the collection sharded? Do we need a
Load Balancer or just point to one box and run the new query? Or would it
be better to hit the LB in case one machine is no longer good to go?

http://loadbalancer:8983/solr/collectionname/select?q=*:*

What features would not yet be ready for sharded setups with SolrCloud? In
the past, facet counts were an issue, grouping? stats? as well as IDF for
sorting by scores. i.e. facet.field=specialties. We want the Cardiologist
specialty to have unique numbers across shards. So if shard1 has 4 people
with Cardiology, and shard2 has 2 people with Cardiology, we would want the
number to be 6. We would want facet.sort to work on counts... I guess we
could index another collection for facets and just use 1 machine for that?
But doesn't that defeat the purpose?

What is the best walk thru for SOLR 5.3.1 ?

Looking at https://wiki.apache.org/solr/SolrCloud

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

RptWithGeometrySpatialField

[William Bell]

David, et al.


2 questions.



Is this a good tradeoff for performance?

Also, RptWithGeometrySpatialField improved performance on what type of
searches? Does it support multivalued lat,lon fields?

Does LatLonType support multivalued lat,lon fields or only RPT allows for
this?


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Parallel SQL / calcite adapter

[William Bell]

How is performance on Calcite?

On Fri, Nov 20, 2015 at 5:12 AM, Joel Bernstein  wrote:

> After reading https://calcite.apache.org/docs/tutorial.html, I think it
> should be possible to use the Solr's JDBC Driver with Calcites JDBC
> adapter.
>
> If you give it a try and run into any problems, please create a jira.
>
> Joel Bernstein
> http://joelsolr.blogspot.com/
>
> On Thu, Nov 19, 2015 at 7:58 PM, Joel Bernstein 
> wrote:
>
> > It's an interesting question. The JDBC driver is still very basic. It
> > would depend on how much of the JDBC spec needs to be implemented to
> > connect to Calcite/Drill.
> >
> > Joel Bernstein
> > http://joelsolr.blogspot.com/
> >
> > On Thu, Nov 19, 2015 at 3:28 AM, Kai Gülzau 
> wrote:
> >
> >>
> >> We are currently evaluating calcite as a SQL facade for different Data
> >> Sources
> >>
> >> -  JDBC
> >>
> >> -  REST
> >>
> >> >SOLR
> >>
> >> -  ...
> >>
> >> I didn't found a "native" calcite adapter for solr (
> >> http://calcite.apache.org/docs/adapter.html).
> >>
> >> Is it a good idea to use the parallel sql feature (over jdbc) to connect
> >> calcite (or apache drill) to solr?
> >> Any suggestions?
> >>
> >>
> >> Thanks,
> >>
> >> Kai Gülzau
> >>
> >
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: fl=value equals?

[William Bell]

How about we just add a new function called equals() and put into the
solution?

On Fri, Nov 13, 2015 at 11:36 AM, simon <mtnes...@gmail.com> wrote:

> Please do push your script to github - I (re)-compile custom code
> infrequently and never remember how to setup the environment.
>
> On Thu, Nov 12, 2015 at 5:14 AM, Upayavira <u...@odoko.co.uk> wrote:
>
> > Okay, makes sense. As to your question - making a new ValueSourceParser
> > that handles 'equals' sounds pretty straight-forward.
> >
> > If it helps, I have somewhere an Ant project that will unpack Solr and
> > compile custom components against it. I could push that to github or
> > something.
> >
> > Upayavira
> >
> > On Thu, Nov 12, 2015, at 07:59 AM, billnb...@gmail.com wrote:
> > > fl=$b tells me it works. Or I can do a sort=$b asc
> > >
> > > The idea is to calculate a score but only include geo if it is not a
> > > national search. Do we want to send in a parameter into the QT which
> > > allows us to omit geo from national searches
> > >
> > >
> > > Bill Bell
> > > Sent from mobile
> > >
> > > > On Nov 11, 2015, at 1:15 AM, Upayavira <u...@odoko.co.uk> wrote:
> > > >
> > > > I concur with Jan - what does b= do?
> > > >
> > > > Also asking, how did you identify that it worked?
> > > >
> > > > Upayavira
> > > >
> > > >> On Wed, Nov 11, 2015, at 02:58 AM, William Bell wrote:
> > > >> I was able to get it to work kinda with a map().
> > > >>
> > > >> http://localhost:8983/solr/select?q=*:*=1=
> > > >> <
> >
> http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,%27national%27),0,geodist())
> > >
> > > >> map($radius,1,1,0,geodist())
> > > >>
> > > >> Where 1= National
> > > >>
> > > >> Do you have an example of a SearchComponent? It would be pretty easy
> > to
> > > >> copy map() and develop an equals() right?
> > > >>
> > > >> if(equals($radius, 'national'), 0, geodist())
> > > >>
> > > >> This would probably be useful for everyone.
> > > >>
> > > >> On Tue, Nov 10, 2015 at 4:05 PM, Jan Høydahl <jan@cominvent.com
> >
> > > >> wrote:
> > > >>
> > > >>> Where is your “b” parameter used? I think that instead of trying to
> > set a
> > > >>> new “b” http param (which solr will not evaluate as a function),
> you
> > should
> > > >>> instead try to insert your function or switch qParser directly
> where
> > the
> > > >>> “b” param is used, e.g. in a bq or similar.
> > > >>>
> > > >>> A bit heavy weight, but you could of course write a custom
> > SearchComponent
> > > >>> to construct your “b” parameter...
> > > >>>
> > > >>> --
> > > >>> Jan Høydahl, search solution architect
> > > >>> Cominvent AS - www.cominvent.com
> > > >>>
> > > >>>> 10. nov. 2015 kl. 23.52 skrev William Bell <billnb...@gmail.com>:
> > > >>>>
> > > >>>> We are trying to look at a value, and change another value based
> on
> > that.
> > > >>>>
> > > >>>> For example, for national search we want to pass in
> > radius=national, and
> > > >>>> then set another variable equal to 0, else set the other variable
> =
> > to
> > > >>>> geodist() calculation.
> > > >>>>
> > > >>>> We tried {!switch} but this only appears to work on fq/q. There is
> > no
> > > >>>> function for constants for equals
> > > >>>
> >
> http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())
> > > >>>>
> > > >>>> This does not work:
> > > >>>>
> > > >>>>
> http://localhost:8983/solr/select?q=*:*=national={!switch
> > > >>>> case.national=0 default=geodist() v=$radius}
> > > >>>>
> > > >>>> Ideas?
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> --
> > > >>>> Bill Bell
> > > >>>> billnb...@gmail.com
> > > >>>> cell 720-256-8076
> > > >>
> > > >>
> > > >> --
> > > >> Bill Bell
> > > >> billnb...@gmail.com
> > > >> cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

fl=value equals?

[William Bell]

We are trying to look at a value, and change another value based on that.

For example, for national search we want to pass in radius=national, and
then set another variable equal to 0, else set the other variable = to
geodist() calculation.

We tried {!switch} but this only appears to work on fq/q. There is no
function for constants for equals

http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())

This does not work:

http://localhost:8983/solr/select?q=*:*=national={!switch
case.national=0 default=geodist() v=$radius}

Ideas?



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: fl=value equals?

[William Bell]

I was able to get it to work kinda with a map().

http://localhost:8983/solr/select?q=*:*=1=
<http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,%27national%27),0,geodist())>
map($radius,1,1,0,geodist())

Where 1= National

Do you have an example of a SearchComponent? It would be pretty easy to
copy map() and develop an equals() right?

if(equals($radius, 'national'), 0, geodist())

This would probably be useful for everyone.

On Tue, Nov 10, 2015 at 4:05 PM, Jan Høydahl <jan@cominvent.com> wrote:

> Where is your “b” parameter used? I think that instead of trying to set a
> new “b” http param (which solr will not evaluate as a function), you should
> instead try to insert your function or switch qParser directly where the
> “b” param is used, e.g. in a bq or similar.
>
> A bit heavy weight, but you could of course write a custom SearchComponent
> to construct your “b” parameter...
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> > 10. nov. 2015 kl. 23.52 skrev William Bell <billnb...@gmail.com>:
> >
> > We are trying to look at a value, and change another value based on that.
> >
> > For example, for national search we want to pass in radius=national, and
> > then set another variable equal to 0, else set the other variable = to
> > geodist() calculation.
> >
> > We tried {!switch} but this only appears to work on fq/q. There is no
> > function for constants for equals
> >
> >
> http://localhost:8983/solr/select?q=*:*=national=if(equals($radius,'national'),0,geodist())
> >
> > This does not work:
> >
> > http://localhost:8983/solr/select?q=*:*=national={!switch
> > case.national=0 default=geodist() v=$radius}
> >
> > Ideas?
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Solr Suggester with Geo?

[William Bell]

http://lucidworks.com/blog/solr-suggester/


Wondering if anyone has uses these new techniques with a boost on
geodist() inverted? So the rows that get returned that are closest
need to come back first.


We are still using Edge Grams since we have not figured out how to
boost the results on geo spatial.


Anyone have thoughts?




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Solr Suggester with Geo?

[William Bell]

Yeah we have that working today. But the issue is we want to use
http://lucidworks.com/blog/solr-suggester/

And you cannot do a boost with that right?



On Mon, Nov 9, 2015 at 12:41 PM, Sameer Maggon <sam...@measuredsearch.com>
wrote:

> Have you looked at the Spatial extensions for Solr? If you are indexing
> Lat/Lon along with your documents, you can compute the distance from the
> origin & use that distance as one of the boost factors to affect the score.
> Typically, use cases around that combine the geo score with other factors
> as a pure sort by geo score might not give you the relevant results.
>
> e.g. typing to search for "sushi restaurants" near Santa Monica, CA - you
> might not want "thai restaurants" that are closest to you. (Local Search
> use case)
>
> https://cwiki.apache.org/confluence/display/solr/Spatial+Search
>
> Thanks,
> --
> *Sameer Maggon*
> www.measuredsearch.com <http://measuredsearch.com/>
> Fully Managed Solr-as-a-Service | Solr Consulting | Solr Support
>
>
>
> On Mon, Nov 9, 2015 at 11:18 AM, William Bell <billnb...@gmail.com> wrote:
>
> > http://lucidworks.com/blog/solr-suggester/
> >
> >
> > Wondering if anyone has uses these new techniques with a boost on
> > geodist() inverted? So the rows that get returned that are closest
> > need to come back first.
> >
> >
> > We are still using Edge Grams since we have not figured out how to
> > boost the results on geo spatial.
> >
> >
> > Anyone have thoughts?
> >
> >
> >
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Looking for SOLR consulting help

[William Bell]

Healthgrades is looking for Solr consulting assistance.

Rate is negotiable based on skillsets. $125 - $175/hr

We are flexible on time.

1. Solr 5.x experience
2. Tuning performance
3. Relevancy and Autosuggest experience
4. Move to Solr Cloud
5. Amazon Linux Debian experience
6. Java experience using InteliJ and SOLRJ
7. Review requirements and implement changes to Solr cores/collections

Please email me directly, no agents.

Needed for 3-6 months. Could go full time.


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

any clean test failing

[William Bell]

It is getting stuck on resolve.

ant clean test

SOLR 5.3.1

[ivy:retrieve] retrieve done (5ms)

Overriding previous definition of property "ivy.version"

[ivy:retrieve] no resolved descriptor found: launching default resolve

Overriding previous definition of property "ivy.version"

[ivy:retrieve] using ivy parser to parse
file:/home/solr/src/lucene_solr_5_3_1A/solr/server/ivy.xml

[ivy:retrieve] :: resolving dependencies :: org.apache.solr#
example;work...@hgsolr2devmstr.healthgrades.com

[ivy:retrieve] confs: [logging]

[ivy:retrieve] validate = true

[ivy:retrieve] refresh = false

[ivy:retrieve] resolving dependencies for configuration 'logging'

[ivy:retrieve] == resolving dependencies for org.apache.solr#
example;work...@hgsolr2devmstr.healthgrades.com [logging]

[ivy:retrieve] == resolving dependencies
org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->log4j#log4j;1.2.17
[logging->master]

[ivy:retrieve] default: Checking cache for: dependency: log4j#log4j;1.2.17
{logging=[master]}

[ivy:retrieve] don't use cache for log4j#log4j;1.2.17: checkModified=true

[ivy:retrieve] tried /home/solr/.ivy2/local/log4j/log4j/1.2.17/ivys/ivy.xml

[ivy:retrieve] tried
/home/solr/.ivy2/local/log4j/log4j/1.2.17/jars/log4j.jar

[ivy:retrieve] local: no ivy file nor artifact found for log4j#log4j;1.2.17

[ivy:retrieve] main: Checking cache for: dependency: log4j#log4j;1.2.17
{logging=[master]}

[ivy:retrieve] main: module revision found in cache: log4j#log4j;1.2.17

[ivy:retrieve] found log4j#log4j;1.2.17 in public

[ivy:retrieve] == resolving dependencies
org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->org.slf4j#slf4j-api;1.7.7
[logging->master]

[ivy:retrieve] default: Checking cache for: dependency:
org.slf4j#slf4j-api;1.7.7 {logging=[master]}

[ivy:retrieve] don't use cache for org.slf4j#slf4j-api;1.7.7:
checkModified=true

[ivy:retrieve] tried
/home/solr/.ivy2/local/org.slf4j/slf4j-api/1.7.7/ivys/ivy.xml

[ivy:retrieve] tried
/home/solr/.ivy2/local/org.slf4j/slf4j-api/1.7.7/jars/slf4j-api.jar

[ivy:retrieve] local: no ivy file nor artifact found for
org.slf4j#slf4j-api;1.7.7

[ivy:retrieve] main: Checking cache for: dependency:
org.slf4j#slf4j-api;1.7.7 {logging=[master]}

[ivy:retrieve] main: module revision found in cache:
org.slf4j#slf4j-api;1.7.7

[ivy:retrieve] found org.slf4j#slf4j-api;1.7.7 in public

[ivy:retrieve] == resolving dependencies
org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->org.slf4j#jcl-over-slf4j;1.7.7
[logging->master]

[ivy:retrieve] default: Checking cache for: dependency:
org.slf4j#jcl-over-slf4j;1.7.7 {logging=[master]}

[ivy:retrieve] don't use cache for org.slf4j#jcl-over-slf4j;1.7.7:
checkModified=true

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: any clean test failing

[William Bell]

OK I deleted /home/solr/.ivy2 and it started working.

On Sat, Oct 24, 2015 at 11:57 AM, William Bell <billnb...@gmail.com> wrote:

> It is getting stuck on resolve.
>
> ant clean test
>
> SOLR 5.3.1
>
> [ivy:retrieve] retrieve done (5ms)
>
> Overriding previous definition of property "ivy.version"
>
> [ivy:retrieve] no resolved descriptor found: launching default resolve
>
> Overriding previous definition of property "ivy.version"
>
> [ivy:retrieve] using ivy parser to parse
> file:/home/solr/src/lucene_solr_5_3_1A/solr/server/ivy.xml
>
> [ivy:retrieve] :: resolving dependencies :: org.apache.solr#
> example;work...@hgsolr2devmstr.healthgrades.com
>
> [ivy:retrieve] confs: [logging]
>
> [ivy:retrieve] validate = true
>
> [ivy:retrieve] refresh = false
>
> [ivy:retrieve] resolving dependencies for configuration 'logging'
>
> [ivy:retrieve] == resolving dependencies for org.apache.solr#
> example;work...@hgsolr2devmstr.healthgrades.com [logging]
>
> [ivy:retrieve] == resolving dependencies
> org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->log4j#log4j;1.2.17
> [logging->master]
>
> [ivy:retrieve] default: Checking cache for: dependency: log4j#log4j;1.2.17
> {logging=[master]}
>
> [ivy:retrieve] don't use cache for log4j#log4j;1.2.17: checkModified=true
>
> [ivy:retrieve] tried
> /home/solr/.ivy2/local/log4j/log4j/1.2.17/ivys/ivy.xml
>
> [ivy:retrieve] tried
> /home/solr/.ivy2/local/log4j/log4j/1.2.17/jars/log4j.jar
>
> [ivy:retrieve] local: no ivy file nor artifact found for
> log4j#log4j;1.2.17
>
> [ivy:retrieve] main: Checking cache for: dependency: log4j#log4j;1.2.17
> {logging=[master]}
>
> [ivy:retrieve] main: module revision found in cache: log4j#log4j;1.2.17
>
> [ivy:retrieve] found log4j#log4j;1.2.17 in public
>
> [ivy:retrieve] == resolving dependencies
> org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->org.slf4j#slf4j-api;1.7.7
> [logging->master]
>
> [ivy:retrieve] default: Checking cache for: dependency:
> org.slf4j#slf4j-api;1.7.7 {logging=[master]}
>
> [ivy:retrieve] don't use cache for org.slf4j#slf4j-api;1.7.7:
> checkModified=true
>
> [ivy:retrieve] tried
> /home/solr/.ivy2/local/org.slf4j/slf4j-api/1.7.7/ivys/ivy.xml
>
> [ivy:retrieve] tried
> /home/solr/.ivy2/local/org.slf4j/slf4j-api/1.7.7/jars/slf4j-api.jar
>
> [ivy:retrieve] local: no ivy file nor artifact found for
> org.slf4j#slf4j-api;1.7.7
>
> [ivy:retrieve] main: Checking cache for: dependency:
> org.slf4j#slf4j-api;1.7.7 {logging=[master]}
>
> [ivy:retrieve] main: module revision found in cache:
> org.slf4j#slf4j-api;1.7.7
>
> [ivy:retrieve] found org.slf4j#slf4j-api;1.7.7 in public
>
> [ivy:retrieve] == resolving dependencies
> org.apache.solr#example;work...@hgsolr2devmstr.healthgrades.com->org.slf4j#jcl-over-slf4j;1.7.7
> [logging->master]
>
> [ivy:retrieve] default: Checking cache for: dependency:
> org.slf4j#jcl-over-slf4j;1.7.7 {logging=[master]}
>
> [ivy:retrieve] don't use cache for org.slf4j#jcl-over-slf4j;1.7.7:
> checkModified=true
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Get this committed

[William Bell]

OK I added the test case.

On Fri, Oct 23, 2015 at 5:05 AM, Shalin Shekhar Mangar <
shalinman...@gmail.com> wrote:

> I can review and commit if you add a test.
>
> On Fri, Oct 23, 2015 at 9:45 AM, William Bell <billnb...@gmail.com> wrote:
> > I can confirm this is working in PROD at 100M hits a day.
> >
> > Can we commit it please? Begging here.
> >
> > https://issues.apache.org/jira/browse/SOLR-7993
> >
> > --
> > Bill Bell
> > billnb...@gmail.com
> > cell 720-256-8076
>
>
>
> --
> Regards,
> Shalin Shekhar Mangar.
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Get this committed

[William Bell]

I can confirm this is working in PROD at 100M hits a day.

Can we commit it please? Begging here.

https://issues.apache.org/jira/browse/SOLR-7993

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: PayloadTermQuery deprecated

[William Bell]

Alan,

Does this code look equivalent? And how do I change PayLoadScoreQuery to do
a Custom Similarity?

PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
AveragePayloadFunction());

@Override
public Query parse() throws SyntaxError {

if (qstr == null || qstr.length() == 0) return null;
//BooleanQuery q = new BooleanQuery();
BooleanQuery.Builder q = new BooleanQuery.Builder();
q.setDisableCoord(true);
if (qstr.length() > 1 && qstr.startsWith("\"") && qstr.endsWith("\"")) {
qstr = qstr.substring(1,qstr.length()-1);
}
String[] nvps = StringUtils.split(qstr, " ");
for (int i = 0; i < nvps.length; i++) {
String[] nv = StringUtils.split(nvps[i], ":");
if (nv.length > 1) {
  if (nv[0].startsWith("+")) {
  SpanTermQuery sq = new SpanTermQuery(new
Term(nv[0].substring(1), nv[1]));
  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
AveragePayloadFunction());
  q.add(psq, Occur.MUST);
//q.add(new PayloadTermQuery(new Term(nv[0].substring(1),
nv[1]), new AveragePayloadFunction(), false), Occur.MUST);
  } else {
//q.add(new PayloadTermQuery(new Term(nv[0], nv[1]), new
AveragePayloadFunction(), false), Occur.SHOULD);
  SpanTermQuery sq = new SpanTermQuery(new Term(nv[0], nv[1]));
  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
AveragePayloadFunction());
  q.add(psq, Occur.SHOULD);
  }
}
}
// return q;
return q.build();
}


On Mon, Oct 19, 2015 at 1:49 AM, Alan Woodward <a...@flax.co.uk> wrote:

> Hi Bill,
>
> This looks like an oversight on my part when migrating the payload scoring
> queries - can you open a JIRA ticket to add 'includeSpanScore' as an option
> to PayloadScoreQuery?
>
> As a workaround, you should be able to use a custom similarity that
> returns 1 for all scores (see IndexSearcher.NON_SCORING_SIMILARITY for an
> implementation that returns 0, you could just clone that and change
> SimScorer.score())
>
> Alan Woodward
> www.flax.co.uk
>
>
> On 19 Oct 2015, at 00:39, William Bell wrote:
>
> > Here is my first stab at it. Thoughts?
> >
> > Question:
> >
> > new PayloadTermQuery(new Term(nv[0].substring(1), nv[1]), new
> > AveragePayloadFunction(), false)
> >
> > How do I handle the "false"  ? It means boolean includeSpanScore
> >
> >
> > @Override
> > public Query parse() throws SyntaxError {
> >
> >if (qstr == null || qstr.length() == 0) return null;
> >//BooleanQuery q = new BooleanQuery();
> >BooleanQuery.Builder q = new BooleanQuery.Builder();
> >q.setDisableCoord(true);
> >if (qstr.length() > 1 && qstr.startsWith("\"") &&
> qstr.endsWith("\"")) {
> >qstr = qstr.substring(1,qstr.length()-1);
> >}
> >String[] nvps = StringUtils.split(qstr, " ");
> >for (int i = 0; i < nvps.length; i++) {
> >String[] nv = StringUtils.split(nvps[i], ":");
> >if (nv.length > 1) {
> >  if (nv[0].startsWith("+")) {
> >  SpanTermQuery sq = new SpanTermQuery(new
> > Term(nv[0].substring(1), nv[1]));
> >  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
> > AveragePayloadFunction());
> >  q.add(psq, Occur.MUST);
> >//q.add(new PayloadTermQuery(new Term(nv[0].substring(1),
> > nv[1]), new AveragePayloadFunction(), false), Occur.MUST);
> >  } else {
> >//q.add(new PayloadTermQuery(new Term(nv[0], nv[1]), new
> > AveragePayloadFunction(), false), Occur.SHOULD);
> >  SpanTermQuery sq = new SpanTermQuery(new Term(nv[0],
> nv[1]));
> >  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
> > AveragePayloadFunction());
> >  q.add(psq, Occur.SHOULD);
> >  }
> >}
> >}
> >// return q;
> >return q.build();
> > }
> >
> >
> > On Sun, Oct 18, 2015 at 4:46 PM, William Bell <billnb...@gmail.com>
> wrote:
> >
> >> Wondering how to change my payload based on example:
> >>
> >>
> https://lucidworks.com/blog/2014/06/13/end-to-end-payload-example-in-solr/
> >>
> >> PayloadTermQuery and BooleanQuery are deprecated in 5.3.x
> >>
> >> @Override
> >> public Query parse() throws SyntaxError {
> >>
> >>if (qstr == null || qstr.length() == 0) return null;
> >>BooleanQuery q = new BooleanQuery()

Re: PayloadTermQuery deprecated

[William Bell]

Here is my first stab at it. Thoughts?

Question:

new PayloadTermQuery(new Term(nv[0].substring(1), nv[1]), new
AveragePayloadFunction(), false)

How do I handle the "false"  ? It means boolean includeSpanScore


@Override
public Query parse() throws SyntaxError {

if (qstr == null || qstr.length() == 0) return null;
//BooleanQuery q = new BooleanQuery();
BooleanQuery.Builder q = new BooleanQuery.Builder();
q.setDisableCoord(true);
if (qstr.length() > 1 && qstr.startsWith("\"") && qstr.endsWith("\"")) {
qstr = qstr.substring(1,qstr.length()-1);
}
String[] nvps = StringUtils.split(qstr, " ");
for (int i = 0; i < nvps.length; i++) {
String[] nv = StringUtils.split(nvps[i], ":");
if (nv.length > 1) {
  if (nv[0].startsWith("+")) {
  SpanTermQuery sq = new SpanTermQuery(new
Term(nv[0].substring(1), nv[1]));
  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
AveragePayloadFunction());
  q.add(psq, Occur.MUST);
//q.add(new PayloadTermQuery(new Term(nv[0].substring(1),
nv[1]), new AveragePayloadFunction(), false), Occur.MUST);
  } else {
//q.add(new PayloadTermQuery(new Term(nv[0], nv[1]), new
AveragePayloadFunction(), false), Occur.SHOULD);
  SpanTermQuery sq = new SpanTermQuery(new Term(nv[0], nv[1]));
  PayloadScoreQuery psq = new PayloadScoreQuery(sq, new
AveragePayloadFunction());
  q.add(psq, Occur.SHOULD);
  }
    }
}
// return q;
return q.build();
}


On Sun, Oct 18, 2015 at 4:46 PM, William Bell <billnb...@gmail.com> wrote:

> Wondering how to change my payload based on example:
>
> https://lucidworks.com/blog/2014/06/13/end-to-end-payload-example-in-solr/
>
> PayloadTermQuery and BooleanQuery are deprecated in 5.3.x
>
> @Override
> public Query parse() throws SyntaxError {
>
> if (qstr == null || qstr.length() == 0) return null;
> BooleanQuery q = new BooleanQuery();
> if (qstr.length() > 1 && qstr.startsWith("\"") && qstr.endsWith("\"")) {
> qstr = qstr.substring(1,qstr.length()-1);
> }
> String[] nvps = StringUtils.split(qstr, " ");
> for (int i = 0; i < nvps.length; i++) {
> String[] nv = StringUtils.split(nvps[i], ":");
> if (nv.length > 1) {
>   if (nv[0].startsWith("+")) {
> q.add(new PayloadTermQuery(new Term(nv[0].substring(1), nv[1]),
>   new AveragePayloadFunction(), false), Occur.MUST);
>   } else {
> q.add(new PayloadTermQuery(new Term(nv[0], nv[1]),
>   new AveragePayloadFunction(), false), Occur.SHOULD);
>   }
> }
> }
> return q;
> }
>
>
> --
> Bill Bell
> billnb...@gmail.com
> cell 720-256-8076
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Lucene Revolution ?

[William Bell]

Summary of new things in 5.x or 6.x that is catching fire? Ideas? Thoughts
on the Facet caching flame war?


On Sun, Oct 18, 2015 at 3:47 PM, Susheel Kumar <susheel2...@gmail.com>
wrote:

> I couldn't also make it.  Would love to hear more who make it.
>
> Thanks,
> Susheel
>
> On Sun, Oct 18, 2015 at 10:53 AM, Jack Krupansky <jack.krupan...@gmail.com
> >
> wrote:
>
> > Sorry I missed out this year. I thought it was next month and hadn't seen
> > any reminders. Just last Tuesday I finally got around to googling the
> > conference and was shocked to read that it was the next day. Oh well.
> > Personally I'm less interested in the formal sessions than the informal
> > networking.
> >
> > In any case, keep those user reports flowing. I'm sure there are plenty
> of
> > people who didn't make it to the conference.
> >
> > -- Jack Krupansky
> >
> > On Sun, Oct 18, 2015 at 8:52 AM, Erik Hatcher <erik.hatc...@gmail.com>
> > wrote:
> >
> > > The Revolution was not televised (though heavily tweeted, and videos of
> > > sessions to follow eventually).  A great time was had by all.  Much
> > > learning!  Much collaboration. Awesome event if I may say so myself.
> I'm
> > > proud to be a part of the organization that put on the best conference
> > I've
> > > been to to date (til next years Revolution). Don't miss the next one :)
> > >
> > > Re ES/AWS: what about it?   Solr is a first class AWS citizen,
> employing
> > > Solr folks, and certainly where many of our customers deploy their
> > > infrastructure, Solr, Fusion, etc.
> > >
> > >Erik
> > >
> > >
> > >
> > > > On Oct 18, 2015, at 01:02, William Bell <billnb...@gmail.com> wrote:
> > > >
> > > > How did Lucene Revolution 2015 go last week?
> > > >
> > > > Also, what about Amazon's release of Elastic Search as a managed
> > service
> > > in
> > > > AWS?
> > > >
> > > > --
> > > > Bill Bell
> > > > billnb...@gmail.com
> > > > cell 720-256-8076
> > >
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

PayloadTermQuery deprecated

[William Bell]

Wondering how to change my payload based on example:

https://lucidworks.com/blog/2014/06/13/end-to-end-payload-example-in-solr/

PayloadTermQuery and BooleanQuery are deprecated in 5.3.x

@Override
public Query parse() throws SyntaxError {

if (qstr == null || qstr.length() == 0) return null;
BooleanQuery q = new BooleanQuery();
if (qstr.length() > 1 && qstr.startsWith("\"") && qstr.endsWith("\"")) {
qstr = qstr.substring(1,qstr.length()-1);
}
String[] nvps = StringUtils.split(qstr, " ");
for (int i = 0; i < nvps.length; i++) {
String[] nv = StringUtils.split(nvps[i], ":");
if (nv.length > 1) {
  if (nv[0].startsWith("+")) {
q.add(new PayloadTermQuery(new Term(nv[0].substring(1), nv[1]),
  new AveragePayloadFunction(), false), Occur.MUST);
  } else {
q.add(new PayloadTermQuery(new Term(nv[0], nv[1]),
  new AveragePayloadFunction(), false), Occur.SHOULD);
  }
}
}
return q;
}


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Add replication to http://localhost:8983/solr/#/~cores/extinfo

[William Bell]

It would be very useful to add a button to replicate (like we have
optimize) on

http://localhost:8983/solr/#/~cores/extinfo

Also a link to the core would be great too


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Lucene Revolution ?

[William Bell]

How did Lucene Revolution 2015 go last week?

Also, what about Amazon's release of Elastic Search as a managed service in
AWS?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: AutoComplete Feature in Solr

[William Bell]

We want to use suggester but also want to show those results closest to my
lat,long... Kinda combine suggester and bq=geodist()

On Mon, Oct 12, 2015 at 2:24 PM, Salman Ansari 
wrote:

> Hi,
>
> I have been trying to get the autocomplete feature in Solr working with no
> luck up to now. First I read that "suggest component" is the recommended
> way as in the below article (and this is the exact functionality I am
> looking for, which is to autocomplete multiple words)
>
> http://blog.trifork.com/2012/02/15/different-ways-to-make-auto-suggestions-with-solr/
>
> Then I tried implementing suggest as described in the following articles in
> this order
> 1) https://wiki.apache.org/solr/Suggester#SearchHandler_configuration
> 2) http://solr.pl/en/2010/11/15/solr-and-autocomplete-part-2/  (I
> implemented suggesting phrases)
> 3)
>
> http://stackoverflow.com/questions/18132819/how-to-have-solr-autocomplete-on-whole-phrase-when-query-contains-multiple-terms
>
> With no luck, after implementing each article when I run my query as
> http://[MySolr]:8983/solr/entityStore114/suggest?spellcheck.q=Barack
>
>
>
> I get
> 
> 
> 0
> 0
> 
> 
>
>  Although I have an entry for Barack Obama in my index. I am posting my
> Solr configuration as well
>
> 
>  
>   suggest
>   org.apache.solr.spelling.suggest.Suggester
>name="lookupImpl">org.apache.solr.spelling.suggest.fst.FSTLookup
>   entity_autocomplete
> true
>  
> 
>
>   class="org.apache.solr.handler.component.SearchHandler">
>  
>   true
>   suggest
>   10
> true
> false
>  
>  
>   suggest
>  
> 
>
> It looks like a very simple job, but even after following so many articles,
> I could not get it right. Any comment will be appreciated!
>
> Regards,
> Salman
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

??

[William Bell]

http://www.slideshare.net/lucidworks/high-performance-solr-and-jvm-tuning-strategies-used-for-map-quests-search-ahead-darren-spehr

See ArrayBlockingQueue.

What would this help with?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

solr-user@lucene.apache.org

[William Bell]

What should this be set to?

Do you set it with -Dsolr.jetty.https.acceptQueueSize=5000 ?



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

FieldCache?

[William Bell]

So the FieldCache was removed from Solr 5.

What is the implication of this? Should we move all facets to DocValues
when we have high cardinality (lots of values) ? Are we adding it back?

Other ideas to improve performance?

>From Mike M:

FieldCache is gone (moved to a dedicated UninvertingReader in the miscmodule).
This means when you intend to sort on a field, you should index that field
using doc values, which is much faster and less heap consuming than
FieldCache.

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Ideas

[William Bell]

We have some Denial of service attacks on our web site. SOLR threads are
going crazy.

Basically someone is hitting start=15 + and rows=20. The start is crazy
large.

And then they jump around. start=15 then start=213030 etc.

Any ideas for how to stop this besides blocking these IPs?

Sometimes it is Google doing it even though these search results are set
with No-index and No-Follow on these pages.

Thoughts? Ideas?

Thanks

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: solr training

[William Bell]

How about in Denver?

On Sun, Sep 13, 2015 at 7:53 PM, Otis Gospodnetić <
otis.gospodne...@gmail.com> wrote:

> Hi Tim,
>
> A slightly delayed reply ;)
> We are running Solr training in NYC next month -
> http://sematext.com/training/solr-training.html - 2nd seat is 50% off.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, May 1, 2015 at 2:18 PM, Tim Dunphy  wrote:
>
> > Hey guys,
> >
> >  My company has a training budget that it wants me to use. So what I'd
> like
> > to find out is if there is any instructor lead courses in the NY/NJ area,
> > or courses online that are instructor lead that you could recommend?
> >
> > Thanks,
> > Tim
> >
> > --
> > GPG me!!
> >
> > gpg --keyserver pool.sks-keyservers.net --recv-keys F186197B
> >
>



-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: DataImportHandler scheduling

[William Bell]

We should add a simple scheduler in the UI. It is very useful. To schedule
various actions:

- Full index
- Delta Index
- Replicate




On Tue, Sep 1, 2015 at 12:41 PM, Shawn Heisey  wrote:

> On 9/1/2015 11:45 AM, Troy Edwards wrote:
> > My initial thought was to use scheduling built with DIH:
> > http://wiki.apache.org/solr/DataImportHandler#Scheduling
> >
> > But I think just a cron job should do the same for me.
>
> The dataimport scheduler does not exist in any Solr version.  This is a
> proposed feature, with the enhancement issue open for more than four years:
>
> https://issues.apache.org/jira/browse/SOLR-2305
>
> I have updated the wiki page to state the fact that the scheduler is a
> proposed improvement, not a usable feature.
>
> Thanks,
> Shawn
>
>


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Query timeAllowed and its behavior.

[William Bell]

As we reported, we are having issues with timeAllowed on 5.2.1. If we set a
timeAllowed=1 and then run the same query with timeAllowed=3 we get the
# of rows that was returned on the first query.

It appears the results are cached when exceeding the timeAllowed, like the
results are correct - when they are truncated.

SEEMS LIKE A BUG TO ME.

On Tue, Aug 25, 2015 at 5:16 AM, Jonathon Marks (BLOOMBERG/ LONDON) 
jmark...@bloomberg.net wrote:

 timeAllowed applies to the time taken by the collector in each shard
 (TimeLimitingCollector). Once timeAllowed is exceeded the collector
 terminates early, returning any partial results it has and freeing the
 resources it was using.
 From Solr 5.0 timeAllowed also applies to the query expansion phase and
 SolrClient request retry.

 From: solr-user@lucene.apache.org At: Aug 25 2015 10:18:07
 Subject: Re:Query timeAllowed and its behavior.

 Hi,

 Kindly help me understand the query time allowed attribute. The following
 is set in solrconfig.xml.
 int name=timeAllowed30/int

 Does this setting stop the query from running after the timeAllowed is
 reached? If not is there a way to stop it as it will occupy resources in
 background for no benefit.

 Thanks,
 Modassar





-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

SOLR 5.3

[William Bell]

At lucene.apache.org/solr it says SOLR 5.3 is there, but when I click on
downloads it shows Solr 5.2.1... ??

APACHE SOLR™ 5.3.0Solr is the popular, blazing-fast, open source
enterprise search platform built on Apache Lucene™.

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: DIH delta-import pk

[William Bell]

Send the SQL and Schema.xml. Also logs. Does it complain about _id_ or you
field in schema?




On Sun, Aug 23, 2015 at 4:55 AM, CrazyDiamond crazy_diam...@mail.ru wrote:

 Now  I set db id as unique field and uuid field,which should be generated
 automatically as required. but when i add document i have an error that my
 required uuid field is missing.



 --
 View this message in context:
 http://lucene.472066.n3.nabble.com/DIH-delta-import-pk-tp4224342p4224701.html
 Sent from the Solr - User mailing list archive at Nabble.com.




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

jetty.xml

[William Bell]

We sometimes get a spike in Solr, and we get like 3K of threads and then
timeouts...

In Solr 5.2.1 the defult jetty settings is kinda crazy for threads - since
the value is HIGH!

What do others recommend?

Fusion jetty settings for Threads:


Get name=ThreadPool

  Set name=minThreads type=intProperty name=threads.min
default=10//Set

*  Set name=maxThreads type=intProperty name=threads.max
default=200//Set*

  Set name=idleTimeout type=intProperty name=threads.timeout
default=*6*//Set

  Set name=detailedDumpfalse/Set

/Get



This is better than SOLR default:


  Get name=ThreadPool

Set name=minThreads type=intProperty
name=solr.jetty.threads.min default=10//Set

Set name=maxThreads type=intProperty
name=solr.jetty.threads.max default=1//Set

Set name=idleTimeout type=intProperty
name=solr.jetty.threads.idle.timeout default=5000//Set

Set name=stopTimeout type=intProperty
name=solr.jetty.threads.stop.timeout default=6//Set

Set name=detailedDumpfalse/Set

  /Get

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

SOLR 5.3

[William Bell]

When do we think SOLR 5.3 might come out?

It is nearly ready?

-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Why is /query needed for Json Facet?

[William Bell]

I tried using /select and this query does not work? Cannot understand why.
Passing Parameters via JSON

We can also pass normal request parameters in the JSON body within the
params block:
$ curl http://localhost:8983/solr/query?fl=title,author-d '
{
  params:{
q:title:hero,
rows:1
  }
}
'

Which is equivalent to:
$ curl http://localhost:8983/solr/query?fl=title
,authorq=title:herorows=1


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Why is /query needed for Json Facet?

[William Bell]

OK I figured it out. The documentation is not updated. The default
component are as follows:

FacetModule.COMPONENT_NAME = facet_module

Thus. The following is the default with the new facet_module.

We need someone to update the solrconfig.xml and the docs.

arr name=components

  strquery/str

  strfacet/str

 strfacet_module/str

  strmlt/str

  strhighlight/str

  strstats/str

  strdebug/str

  strexpand/str

/arr

 protected ListString getDefaultComponents()

  {

ArrayListString names = new ArrayList(6);

names.add( QueryComponent.COMPONENT_NAME );

names.add( FacetComponent.COMPONENT_NAME );

names.add( FacetModule.COMPONENT_NAME );

names.add( MoreLikeThisComponent.COMPONENT_NAME );

names.add( HighlightComponent.COMPONENT_NAME );

names.add( StatsComponent.COMPONENT_NAME );

names.add( DebugComponent.COMPONENT_NAME );

names.add( ExpandComponent.COMPONENT_NAME);

return names;

  }



On Mon, Aug 3, 2015 at 11:31 AM, William Bell billnb...@gmail.com wrote:

 I tried using /select and this query does not work? Cannot understand why.
 Passing Parameters via JSON

 We can also pass normal request parameters in the JSON body within the
 params block:
 $ curl http://localhost:8983/solr/query?fl=title,author-d '
 {
   params:{
 q:title:hero,
 rows:1
   }
 }
 '

 Which is equivalent to:
 $ curl http://localhost:8983/solr/query?fl=title
 ,authorq=title:herorows=1


 --
 Bill Bell
 billnb...@gmail.com
 cell 720-256-8076




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Parameterized values

[William Bell]

That would be pretty bizarre.

I'll try it.

On Wed, Jul 29, 2015 at 1:00 PM, Mikhail Khludnev 
mkhlud...@griddynamics.com wrote:

 hm. Did you try
 str name=pspecPS127/str
 str name=hqval1hosp_quality_spec_boost:${pspec:${pspec}}/str
 ?


 On Tue, Jul 28, 2015 at 8:16 PM, William Bell billnb...@gmail.com wrote:

  http://yonik.com/solr-query-parameter-substitution/
 
  This is not working as part of QTs.
 
  Cannot load the core, since ${value} is being used for XML parameters for
  system property substitution.
 
  https://wiki.apache.org/solr/SolrConfigXml#System_property_substitution
 
  Can we support both?
 
  str name=pspecPS127/str
  str name=hqval1hosp_quality_spec_boost:${pspec}/str
 
 
  This does not work.
 
 
  --
  Bill Bell
  billnb...@gmail.com
  cell 720-256-8076
 



 --
 Sincerely yours
 Mikhail Khludnev
 Principal Engineer,
 Grid Dynamics

 http://www.griddynamics.com
 mkhlud...@griddynamics.com




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Parameterized values

[William Bell]

http://yonik.com/solr-query-parameter-substitution/

This is not working as part of QTs.

Cannot load the core, since ${value} is being used for XML parameters for
system property substitution.

https://wiki.apache.org/solr/SolrConfigXml#System_property_substitution

Can we support both?

str name=pspecPS127/str
str name=hqval1hosp_quality_spec_boost:${pspec}/str


This does not work.


-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Parameterized values

[William Bell]

https://issues.apache.org/jira/browse/SOLR-7846

On Tue, Jul 28, 2015 at 11:16 AM, William Bell billnb...@gmail.com wrote:

 http://yonik.com/solr-query-parameter-substitution/

 This is not working as part of QTs.

 Cannot load the core, since ${value} is being used for XML parameters for
 system property substitution.

 https://wiki.apache.org/solr/SolrConfigXml#System_property_substitution

 Can we support both?

 str name=pspecPS127/str
 str name=hqval1hosp_quality_spec_boost:${pspec}/str


 This does not work.


 --
 Bill Bell
 billnb...@gmail.com
 cell 720-256-8076




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076

Re: Unexpected docvalues type error using result grouping - Use UninvertingReader or index with docvalues

[William Bell]

You could try stopping SOLR, going into the data directory and rm -rf * and
starting SOLR again.

Did you use the schema REST api? Residual ?

On Thu, Jul 23, 2015 at 6:57 PM, Shamik Bandopadhyay sham...@gmail.com
wrote:

 Hi,

  I'm facing this weird error while running result grouping queries. This
 started when I turned on docvalues for an existing facet field and
 indexed the documents. Looking at the exception, I reverted back the change
 and re-indexed the documents again. But I'm still getting the exception,
 here's the stack trace.

 ERROR SolrCore org.apache.solr.common.SolrException: Exception during
 facet.field: MediaType

 org.apache.solr.common.SolrException: Exception during facet.field:
 MediaType
 at
 org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:596)
 at
 org.apache.solr.request.SimpleFacets$3.call(SimpleFacets.java:581)
 at java.util.concurrent.FutureTask.run(FutureTask.java:266)
 at
 org.apache.solr.request.SimpleFacets$2.execute(SimpleFacets.java:535)
 at
 org.apache.solr.request.SimpleFacets.getFacetFieldCounts(SimpleFacets.java:606)
 at
 org.apache.solr.request.SimpleFacets.getFacetCounts(SimpleFacets.java:258)
 at
 org.apache.solr.handler.component.FacetComponent.process(FacetComponent.java:107)
 at
 org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:227)
 at
 org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:144)
 at org.apache.solr.core.SolrCore.execute(SolrCore.java:2006)
 at
 org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:777)
 at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413)
 at
 org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:204)
 at
 org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
 at
 org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
 at
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
 at
 org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:557)
 at
 org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
 at
 org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
 at
 org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
 at
 org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
 at
 org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
 at
 org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
 at
 org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
 at
 org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
 at
 org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
 at org.eclipse.jetty.server.Server.handle(Server.java:368)
 at
 org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
 at
 org.eclipse.jetty.server.BlockingHttpConnection.handleRequest(BlockingHttpConnection.java:53)
 at
 org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
 at
 org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
 at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
 at
 org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
 at
 org.eclipse.jetty.server.BlockingHttpConnection.handle(BlockingHttpConnection.java:72)
 at
 org.eclipse.jetty.server.bio.SocketConnector$ConnectorEndPoint.run(SocketConnector.java:264)
 at
 org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
 at
 org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
 at java.lang.Thread.run(Thread.java:745)
 Caused by: java.lang.IllegalStateException: unexpected docvalues type
 NONE for field 'MediaType' (expected one of [SORTED, SORTED_SET]). Use
 UninvertingReader or index with docvalues.
 at org.apache.lucene.index.DocValues.checkField(DocValues.java:208)
 at
 org.apache.lucene.index.DocValues.getSortedSet(DocValues.java:306)
 at
 org.apache.lucene.search.grouping.term.TermGroupFacetCollector$MV.doSetNextReader(TermGroupFacetCollector.java:282)
 at
 org.apache.lucene.search.SimpleCollector.getLeafCollector(SimpleCollector.java:33)
 at
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:583)
 at
 org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:284)
 at
 

Re: LFU vs solr.FastLRUCache

[William Bell]

We should get that done!! :)

On Mon, Jul 6, 2015 at 7:09 AM, Shawn Heisey apa...@elyograg.org wrote:

 On 7/5/2015 10:11 PM, William Bell wrote:
  Has anyone used  solr.LFUCache in Production to replace:
 
  filterCache class=solr.FastLRUCache
 
size=4096
 
initialSize=4096 cleanupThread=true
autowarmCount=32/
 
  Thoughts?

 I wrote the LFUCache.

 It's the most basic and naive implementation possible, so when an entry
 must be evicted, it has to loop through every single entry in the cache
 to see how frequently it has been used, making it fairly inefficient
 over the long term unless the cache size is very small.

 It would not be a good idea to use it for a large cache, which I think
 size 4096 would definitely be.

 There is a more efficient implementation that I also wrote, but I
 haven't had the time to devote to final touches before getting it
 committed.

 https://issues.apache.org/jira/browse/SOLR-3393

 Thanks,
 Shawn




-- 
Bill Bell
billnb...@gmail.com
cell 720-256-8076