Re: Filter query with multiple raw/literal ORs

2014-04-03 Thread Yonik Seeley
On Thu, Apr 3, 2014 at 7:04 PM, Gregory Chanan gcha...@cloudera.com wrote: {!raw f=foobar v=*} OR {!raw f=foobar v=a} but this seems to give me the same results as the single-clause query. Right... the query parser is detected as raw and the rest of the value is used as the term (or rather it's

Re: Filter caching

2014-03-31 Thread Yonik Seeley
On Mon, Mar 31, 2014 at 2:43 PM, youknow...@heroicefforts.net youknow...@heroicefforts.net wrote: Re-reading the documentation, it seems that Solr caches the results of the fq parameter, not lower level field constraints. This would imply that breaking a single complex boolean filter into

Re: facet doesnt display all possibilities after selecting one

2014-03-27 Thread Yonik Seeley
On Thu, Mar 27, 2014 at 8:56 AM, Andreas Owen ao...@swissonline.ch wrote: when i select a facet in thema_f all the others in the group disapear OK, I see you're excluding filters tagged with thema_f when faceting on the thema_f field. str

Re: document level security filter solution for Solr

2014-03-25 Thread Yonik Seeley
Depending on requirements, another option for simple security is to store the security info in the index and utilize a join. This really only works when you have a single shard since joins aren't distributed. # the documents, with permissions id:doc1, perms:public,... id:doc2, perms:group1

Re: join and filter query with AND

2014-03-21 Thread Yonik Seeley
Correct. This is only a limitation of embedding a local-params style subquery within lucene syntax. The parser, not knowing the syntax of the embedded query, currently assumes the query text ends at whitespace or other special punctuation such as ). Original: (({!join from=inner_id to=outer_id

Re: Difference between addfield and setfield in SolrInputDocument

2014-03-17 Thread Yonik Seeley
On Mon, Mar 17, 2014 at 10:22 AM, vit bulgako...@yahoo.com wrote: Could someone explain me, please, the difference between addfield and setfield in SolrInputDocument addField will add another value to any existing values for the field. setField will just overwrite anything that is already

Re: Deep paging in parallel with solr cloud - OutOfMemory

2014-03-17 Thread Yonik Seeley
On Mon, Mar 17, 2014 at 7:14 PM, Greg Pendlebury greg.pendleb...@gmail.com wrote: My suspicion is that it won't work in parallel Deep paging with cursorMark does work with distributed search (assuming that's what you meant by parallel... querying sub-shards in parallel?). -Yonik

Re: example schema now stores most field values

2014-03-15 Thread Yonik Seeley
Perhaps so atomic updates work? -Yonik http://heliosearch.org - solve Solr GC pauses with off-heap filters and fieldcache On Sat, Mar 15, 2014 at 1:02 PM, Michael Sokolov msoko...@safaribooksonline.com wrote: While upgrading from 4.2.1 to 4.6.1 I noticed that many of the fields defined in the

Re: [ANNOUNCE] Heliosearch 0.04

2014-03-15 Thread Yonik Seeley
On Fri, Mar 14, 2014 at 1:11 PM, Yonik Seeley yo...@heliosearch.com wrote: I've been meaning to create a heliosearch 4x branch to get us a big step closer to cutting a stable release. I should probably do that sooner rather than later... Done. I've created branch helio which is based

Re: [ANNOUNCE] Heliosearch 0.04

2014-03-14 Thread Yonik Seeley
term plans for a more stable release that would be advisable for production use? --Mike On Mon, Mar 10, 2014 at 1:04 PM, Yonik Seeley yo...@heliosearch.com wrote: Changes from the previous release are primarily off-heap FieldCache support for strings as well as as all numerics (the previous

Re: [ANNOUNCE] Heliosearch 0.04

2014-03-14 Thread Yonik Seeley
for a more stable release that would be advisable for production use? --Mike On Mon, Mar 10, 2014 at 1:04 PM, Yonik Seeley yo...@heliosearch.com wrote: Changes from the previous release are primarily off-heap FieldCache support for strings as well as as all numerics (the previous release

Re: [ANNOUNCE] Heliosearch 0.04

2014-03-14 Thread Yonik Seeley
On Fri, Mar 14, 2014 at 12:52 PM, Alexei Martchenko ale...@martchenko.com.br wrote: Chrome on Windows reports the latest Heliosearch as probable malware Heh. Anyone know how to fix that? I tried downloading with crome on my mac just now and it worked fine. -Yonik http://heliosearch.org -

Re: [ANNOUNCE] Heliosearch 0.04

2014-03-14 Thread Yonik Seeley
On Fri, Mar 14, 2014 at 1:11 PM, Yonik Seeley yo...@heliosearch.com wrote: I've been meaning to create a heliosearch 4x branch to get us a big step closer to cutting a stable release. I should probably do that sooner rather than later... https://github.com/Heliosearch/heliosearch/issues/9

[ANNOUNCE] Heliosearch 0.04

2014-03-10 Thread Yonik Seeley
Changes from the previous release are primarily off-heap FieldCache support for strings as well as as all numerics (the previous release only had integer support). Benchmarks for string fields here: http://heliosearch.org/hs-solr-off-heap-fieldcache-performance Try it out here:

Re: Multiple fq parameters are not executed

2014-03-10 Thread Yonik Seeley
Solr has extensive filtering tests. The first step would be to double check that you see what you think you are seeing, and then try and create an example to reproduce it. For example, this works fine with the example data, and is of the same form as your query: http://localhost:8983/solr/query

Re: Updated to v4.7 - Getting Search requests cannot accept content streams

2014-03-10 Thread Yonik Seeley
I get a different error (but related to the same issue I guess) with the following simple query: /opt/code/heliosearch/solr$ curl -XPOST http://localhost:8983/solr/select?q=*:*; ?xml version=1.0 encoding=UTF-8? response lst name=errorstr name=msgMust specify a Content-Type header with POST

Re: volatile write to make isCleaning visible at ConcurrentLRUCache

2014-03-08 Thread Yonik Seeley
On Sat, Mar 8, 2014 at 2:33 PM, Furkan KAMACI furkankam...@gmail.com wrote: ConcurrentLRUCacheK,V class has that lines: ... long oldestEntry = this.oldestEntry; isCleaning = true; this.oldestEntry = oldestEntry; // volatile write to make isCleaning visible ... What does that

Re: volatile write to make isCleaning visible at ConcurrentLRUCache

2014-03-08 Thread Yonik Seeley
On Sat, Mar 8, 2014 at 3:28 PM, Shawn Heisey s...@elyograg.org wrote: Do we have any idea whether this side effect of volatile access is part of the Java specification Yep, it's part of the JMM (Java Memory Model) and is guaranteed behavior. -Yonik http://heliosearch.org - native off-heap

Re: Fetching uniqueKey and other int quickly from documentCache?

2014-03-03 Thread Yonik Seeley
On Wed, Feb 26, 2014 at 8:53 PM, Yonik Seeley yo...@heliosearch.com wrote: You could try forcing things to go through function queries (via pseudo-fields): fl=field(id), field(myfield) If you're not requesting any stored fields, that *might* currently skip that step. -Yonik http

Re: Fetching uniqueKey and other int quickly from documentCache?

2014-02-26 Thread Yonik Seeley
You could try forcing things to go through function queries (via pseudo-fields): fl=field(id), field(myfield) If you're not requesting any stored fields, that *might* currently skip that step. -Yonik http://heliosearch.org - native off-heap filters and fieldcache for solr On Mon, Feb 24, 2014

Re: DistributedSearch: Skipping STAGE_GET_FIELDS?

2014-02-23 Thread Yonik Seeley
On Sun, Feb 23, 2014 at 1:08 PM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: I should clarify though that this optimization only works with fl=id,score. Although it seems like it should be relatively simple to make it work with other fields as well, by passing down the complete fl

Re: Increasing number of SolrIndexSearcher (Leakage)?

2014-02-18 Thread Yonik Seeley
On Mon, Feb 17, 2014 at 1:34 AM, Nguyen Manh Tien tien.nguyenm...@gmail.com wrote: - *But after i index some docs and run softCommit or hardCommit with openSearcher=false, number of SolrIndexSearcher increase by 1* This is fine... it's more of an internal implementation detail (we open what is

Re: block join and atomic updates

2014-02-13 Thread Yonik Seeley
On Thu, Feb 13, 2014 at 8:25 AM, m...@preselect-media.com wrote: Is there any workaround to perform atomic updates on blocks or do I have to re-index the parent document and all its children always again if I want to update a field? The latter, unfortunately. -Yonik http://heliosearch.org -

Re: Problem querying large StrField?

2014-02-10 Thread Yonik Seeley
On Mon, Feb 10, 2014 at 12:42 AM, Luis Lebolo luis.leb...@gmail.com wrote: For this to work, we concatenated all the id's into a single comma delimited value. It doesn't sound like you need the resulting big value to be indexed. All you need to do is retrieve it relatively quickly and do your

Re: geofilt customization

2014-02-06 Thread Yonik Seeley
On Thu, Feb 6, 2014 at 5:13 AM, Sohan Kalsariya sohankalsar...@gmail.com wrote: So in short i want to ask that Is it possible not to define the `distance` parameter ? So I can get my desired results? No, the distance parameter is required for geofilt. If you are still looking to calculate

Re: Problem querying large StrField?

2014-02-05 Thread Yonik Seeley
On Wed, Feb 5, 2014 at 1:04 PM, Luis Lebolo luis.leb...@gmail.com wrote: Update: It seems I get the bad behavior (no documents returned) when the length of a value in the StrField is greater than or equal to 32,767 (2^15). Is this some type of bit overflow somewhere? I believe that's the

Re: Lowering query time

2014-02-04 Thread Yonik Seeley
On Tue, Feb 4, 2014 at 12:12 PM, Joel Cohen joel.co...@bluefly.com wrote: I'm trying to get the query time down to ~15 msec. Anyone have any tuning recommendations? I guess it depends on what the slowest part of the query currently is. If you are faceting, it's often that. Also, it's often a

Re: Lowering query time

2014-02-04 Thread Yonik Seeley
On Tue, Feb 4, 2014 at 1:43 PM, Joel Cohen joel.co...@bluefly.com wrote: 1. We are faceting. I'm not a developer so I'm not quite sure how we're doing it. How can I measure? Add debugQuery=true to the request and look at the timings of various components. 2. I'm not sure how we'd force this

[ANN] Heliosearch 0.03 with off-heap field cache

2014-02-03 Thread Yonik Seeley
A new Heliosearch pre-release has been cut for people to try out: https://github.com/Heliosearch/heliosearch/releases Release Notes: - This is Heliosearch v0.03 Heliosearch is forked from Apache Solr and

Re: Order of values in stored field

2014-01-15 Thread Yonik Seeley
On Wed, Jan 15, 2014 at 10:55 AM, Jamie Johnson jej2...@gmail.com wrote: Is there any guarantee in Solr/Lucene for the order of values in a stored field? Yes, order is preserved. -Yonik http://heliosearch.org -- off-heap filters for solr

Re: Splitting strings in Java - how to escape delimiter characters?

2014-01-14 Thread Yonik Seeley
Look at the StrUtils.splitSmart methods... the first variant treats quotes specially, the second variant doesn't (that's the one you probably want). -Yonik http://heliosearch.org -- off-heap filters for solr On Tue, Jan 14, 2014 at 10:07 AM, Shawn Heisey s...@elyograg.org wrote: I have a Java

Re: ANN: Solr Next

2014-01-13 Thread Yonik Seeley
Update on the my initial performance findings for off-heap filters: http://heliosearch.org/off-heap-filters/ -Yonik http://heliosearch.org -- making solr shine On Tue, Jan 7, 2014 at 1:53 PM, Yonik Seeley ysee...@gmail.com wrote: Off-Heap Filters: JVMs have never been good at dealing

Re: ANN: Solr Next

2014-01-13 Thread Yonik Seeley
format can get the comparable gain without changes in design? https://issues.apache.org/jira/browse/LUCENE-5052 On Mon, Jan 13, 2014 at 9:15 PM, Yonik Seeley ysee...@gmail.com wrote: Update on the my initial performance findings for off-heap filters: http://heliosearch.org/off-heap-filters

ANN: Solr Next

2014-01-07 Thread Yonik Seeley
It's time to start working on the next major evolution of Solr (much as we did years ago for the SolrCloud effort). To kick things off, I've started a project on github and implemented off-heap filters, as a first step toward taking performance to the next level. For a number of reasons, we felt

Re: config JoinQParserPlugin

2013-12-30 Thread Yonik Seeley
On Mon, Dec 30, 2013 at 2:43 PM, Ray Cheng rch...@rocketmail.com wrote: Hi, I have trouble configuring JoinQParserPlugin in Solr. I'm using solr-4.6.0. I put this line (and only this line) to solrconfig.xml following Query Parsers comments of solrconfig.xml (solr-4.6.0): !-- example of

Re: config JoinQParserPlugin

2013-12-30 Thread Yonik Seeley
Does it mean the following line I added to solrconfig.xml is not needed and caused Solr start error? Yes. Join is a builtin parser and you should not add any config for it. Most likely that is what is causing your startup error (although it's not entirely clear why it's causing it to fail).

Re: faceted search field sorting

2013-12-15 Thread Yonik Seeley
If you don't want the facets sorted by decreasing count (the default), you can specify facet.sort=index to get index order ((lexicographic by indexed term) -Yonik http://heliosearch.com -- making solr shine On Sun, Dec 15, 2013 at 2:08 PM, MC videm...@gmail.com wrote: Hello, Here is a public

HDS - free tomcat based solr distro

2013-12-10 Thread Yonik Seeley
We've created a Solr distribution called HDS (Heliosearch Distribution for Solr) as the basis for our support + bug-backport subscriptions (but you can use it without one). Since it's both open source and free to use, I figured it would be of general interest here, especially for Tomcat fans.

Re: post filtering for boolean filter queries

2013-12-05 Thread Yonik Seeley
On Thu, Dec 5, 2013 at 7:39 AM, Dmitry Kan solrexp...@gmail.com wrote: Thanks Erick! To be sure we are using cost 101 and no cache. It seems to affect on searches as we expected. Basically with cache on we see more fat spikes around commit points, as cache is getting flushed (we don't rerun

Re: post filtering for boolean filter queries

2013-12-03 Thread Yonik Seeley
On Tue, Dec 3, 2013 at 4:45 AM, Dmitry Kan solrexp...@gmail.com wrote: ok, we were able to confirm the behavior regarding not caching the filter query. It works as expected. It does not cache with {!cache=false}. We are still looking into clarifying the cost assignment: i.e. whether it works

Re: facet method=enum and uninvertedfield limitations

2013-11-14 Thread Yonik Seeley
On Thu, Nov 14, 2013 at 12:03 PM, Lemke, Michael SZ/HZA-ZSW lemke...@schaeffler.com wrote: I am running into performance problems with faceted queries. If I do a q=wordfacet.field=CONTENTfacet=truefacet.limit=10facet.mincount=1facet.method=fcfacet.prefix=arows=0 I am getting an exception:

Re: Sorting memory-efficiently by any numeric field (dates too?)

2013-11-12 Thread Yonik Seeley
For a reasonable top-N, the space efficiency should still be the same as it is really just dominated by the FieldCache representation (is it in-memory or disk-docvalue based). Directly sorting on that numeric field vs deriving a score from the field and sorting on that shouldn't really be that

Re: Sorting memory-efficiently by any numeric field (dates too?)

2013-11-12 Thread Yonik Seeley
On Tue, Nov 12, 2013 at 7:01 PM, Erick Erickson erickerick...@gmail.com wrote: Yonik: Of course I'm not really up on the details of sorting, but aren't there various control structures that are allocated for a sort but not for scoring? I'm thinking of long[maxDoc] type structures in addition

Re: Function query matching

2013-11-11 Thread Yonik Seeley
On Mon, Nov 11, 2013 at 11:39 AM, Peter Keegan peterlkee...@gmail.com wrote: fq=$qq What is the proper syntax? fq={!query v=$qq} -Yonik http://heliosearch.com -- making solr shine

Re: How to cancel a collection 'optimize'?

2013-11-11 Thread Yonik Seeley
On Mon, Nov 11, 2013 at 11:28 AM, Hoggarth, Gil gil.hogga...@bl.uk wrote: I could stop the whole Solr service as as yet there's no audience access to it, but might it be left in an incomplete state and thus try to complete optimisation when the service is restarted? Should be fine. Lucene

Re: Unexpected query result

2013-11-08 Thread Yonik Seeley
On Fri, Nov 8, 2013 at 10:33 AM, Patrick Duc patrick_...@yahoo.fr wrote: russia (web OR NOT(russia) russia (web (*:* -russia)) Negative clauses often need something positive to subtract from... so replace NOT russia with (*:* -russia) -Yonik http://heliosearch.com -- making solr shine

Re: Performance Question: 'facets.missing'

2013-11-06 Thread Yonik Seeley
On Wed, Nov 6, 2013 at 12:07 PM, andres and...@octopart.com wrote: I'm debating whether or not to set the 'facets.missing' parameter to true by default when faceting. What is the performance impact of setting 'facets.missing' to true? It really depends on the faceting method. For some

Re: Can't find some fields in solr result

2013-11-04 Thread Yonik Seeley
On Mon, Nov 4, 2013 at 2:19 PM, gohome190 gohome...@gmail.com wrote: I have a database that has about 25 fields for each entry. However, when I do a solr *:* query, I can only see the first 19 fields for each entry. However, I can successfully use the fields that don't show up as queries. So

Re: Can I combine standardtokenizer with solr.WordDelimiterFilterFactory?

2013-11-01 Thread Yonik Seeley
On Fri, Nov 1, 2013 at 11:38 AM, eShard zim...@yahoo.com wrote: I have and ID that consists of two letters and a number. The whole user title looks like this: Lastname, Firstname (LA12345). Now, with my current configuration, I can search for LA12345 and find the user. However, when I type in

Re: Global IDF vs. Routing

2013-10-23 Thread Yonik Seeley
On Wed, Oct 23, 2013 at 9:03 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Seeing so much work being put in routing and seeing the recent questions about the status of global IDF support made me realize, for the first time really, that with people using routing more and more we should

Re: Skipping caches on a /select

2013-10-17 Thread Yonik Seeley
this queryResultCache assumption today. Cheers, Tim On 16/10/13 06:33 PM, Yonik Seeley wrote: On Wed, Oct 16, 2013 at 6:18 PM, Tim Vaillancourtt...@elementspace.com wrote: I am debugging some /select queries on my Solr tier and would like to see if there is a way to tell Solr to skip

Re: Skipping caches on a /select

2013-10-16 Thread Yonik Seeley
On Wed, Oct 16, 2013 at 6:18 PM, Tim Vaillancourt t...@elementspace.com wrote: I am debugging some /select queries on my Solr tier and would like to see if there is a way to tell Solr to skip the caches on a given /select query if it happens to ALREADY be in the cache. Live queries are being

Re: Using split in updateCSV for SolrCloud 4.4

2013-10-10 Thread Yonik Seeley
Perhaps try adding echoParams=all to check that all of the input params are being parsed as expected. -Yonik On Thu, Oct 10, 2013 at 8:10 PM, Utkarsh Sengar utkarsh2...@gmail.com wrote: Didn't help. This is the complete data: https://gist.github.com/utkarsh2012/6927649 (see merchantList

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

2013-10-08 Thread Yonik Seeley
On Tue, Oct 8, 2013 at 6:29 PM, Brett Hoerner br...@bretthoerner.com wrote: I'm curious what the later shard-local bits do, if anything? I have a very large cluster (256 shards) and I'm sending most of my data with a single composite, e.g. 1234!unique_id, but I'm noticing the data is being

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

2013-10-08 Thread Yonik Seeley
On Tue, Oct 8, 2013 at 7:31 PM, Brett Hoerner br...@bretthoerner.com wrote: This is my clusterstate.json: https://gist.github.com/bretthoerner/0098f741f48f9bb51433 And these are my core sizes (note large ones are sorted to the end): https://gist.github.com/bretthoerner/f5b5e099212194b5dff6

Re: What's the purpose of the bits option in compositeId (Solr 4.5)?

2013-10-08 Thread Yonik Seeley
On Tue, Oct 8, 2013 at 8:27 PM, Shawn Heisey s...@elyograg.org wrote: There is also the distrib=false parameter that will cause the request to be handled directly by the core it is sent to rather than being distributed/balanced by SolrCloud. Right - this is probably the best option for

Re: Use function return value for range queries

2013-10-05 Thread Yonik Seeley
frange (function range) is what you are looking for. http://yonik.com/posts/ranges-over-functions-in-solr-1-4/ -Yonik http://lucidworks.com On Fri, Oct 4, 2013 at 10:21 AM, SandroZbinden zbin...@imagic.ch wrote: Is there a way to use the function return value for a range query For example:

Re: Adding Custom Score

2013-10-03 Thread Yonik Seeley
If you're using solr, things may work out of the box without using classes. https://cwiki.apache.org/confluence/display/solr/Function+Queries -Yonik On Thu, Oct 3, 2013 at 3:36 AM, Ankit Kumar ankitthemight...@gmail.com wrote: In Lucene i need to add custom score using function query classes .

Re: Sum function causing error in solr

2013-09-27 Thread Yonik Seeley
On Fri, Sep 27, 2013 at 2:28 AM, Tanu Garg tanugarg2...@gmail.com wrote: tried this as well. but its not working. It's working fine for me. What version of Solr are you using? What does your complete request look like? -Yonik http://lucidworks.com

Re: Sum function causing error in solr

2013-09-25 Thread Yonik Seeley
On Wed, Sep 25, 2013 at 6:40 AM, Tanu Garg tanugarg2...@gmail.com wrote: My field name is kcmeta/bookmark/count You can always use the field function to handle whacky field names... unquoted if it's simple enough, or quoted if it's not. field(kcmeta/bookmark/count) or

Re: Atomic updates with solr cloud in solr 4.4

2013-09-21 Thread Yonik Seeley
) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608) at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543) at java.lang.Thread.run(Thread.java:722) Sesha On Tue, Sep 17, 2013 at 8:24 PM, Yonik Seeley yo...@lucidworks.com wrote

Re: Need help understanding the use cases behind core auto-discovery

2013-09-20 Thread Yonik Seeley
On Fri, Sep 20, 2013 at 11:56 AM, Timothy Potter thelabd...@gmail.com wrote: Trying to add some information about core.properties and auto-discovery in Solr in Action and am at a loss for what to tell the reader is the purpose of this feature. IMO, it was more a removal of unnecessary central

Re: Atomic updates with solr cloud in solr 4.4

2013-09-17 Thread Yonik Seeley
On Tue, Sep 17, 2013 at 10:47 AM, Sesha Sendhil Subramanian seshasend...@indix.com wrote: I am using solr 4.4 in solr cloud configuration. When i try to 'set' a field in a document using the update request handler, I get a 'missing required field' error. Can you show the exact error message

Re: Committing when indexing in parallel

2013-09-15 Thread Yonik Seeley
On Fri, Sep 13, 2013 at 5:21 PM, Phani Chaitanya pvempaty@gmail.com wrote: I'm wondering what happens to commit while we are indexing in parallel in Solr. Are the indexing update requests blocked until the commit finishes ? Nope. The add (updates) and commit can proceed in parallel.

Re: Stop filter changes in Solr = 4.4

2013-09-13 Thread Yonik Seeley
On Fri, Sep 13, 2013 at 1:07 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: AFAIk, enablePositionIncrements=false is deprecated in 4.x but not removed. It will be removed in 5.0 though. Hmmm, I had missed that. Anyone have pointers to an example of what broken means and why it can't

Re: Unexpected value for boolean field in FunctionQuery

2013-09-10 Thread Yonik Seeley
I just tried a simple test with the example data, and things seem to be working fine... I tried this: http://localhost:8983/solr/select ?q=*:* rows=100 fl=id, inStock, if(inStock,10,0) I saw values of 10 when inStock==true and values of 0 when it was missing or explicitly false. Perhaps

Re: Facet Sort with non ASCII Characters

2013-09-09 Thread Yonik Seeley
On Mon, Sep 9, 2013 at 7:16 AM, Sandro Zbinden zbin...@imagic.ch wrote: Is there a plan to add support for alphabetical facet sorting with non ASCII Characters ? The entire unicode range should already work. Can you give an example of what you would like to see? -Yonik http://lucidworks.com

Re: AND not working

2013-08-15 Thread Yonik Seeley
I can reproduce something like this by specifying a field that doesn't exist for a qf param. This seems like a bug... if a field doesn't exist, we should throw an exception (since it's a parameter error not related to the q string where we avoid throwing any errors). -Yonik http://lucidworks.com

Re: Huge discrepancy between QTime and ElapsedTime

2013-08-14 Thread Yonik Seeley
On Wed, Aug 14, 2013 at 12:39 PM, Shawn Heisey s...@elyograg.org wrote: You also have grouping enabled. From what I understand, that can be slow. If you turn that off, what happens to your elapsed times? QTime would include that. It includes everything up until the point where the response

Re: Of tlogs and atomic updates

2013-08-13 Thread Yonik Seeley
On Tue, Aug 13, 2013 at 10:11 AM, Erick Erickson erickerick...@gmail.com wrote: A question recently came up: Does the tlog store the entire document when an atomic update happens or just the incoming delta? My guess is that it stores the entire document, but that's a guess... Correct. -Yonik

Re: Of tlogs and atomic updates

2013-08-13 Thread Yonik Seeley
On Tue, Aug 13, 2013 at 11:01 AM, Anirudha Jadhav aniru...@nyu.edu wrote: quick question on a similar topic, for a NRT call to index a doc ,returns a success return code, if and only if all available server have successfully written the doc to their tlog. correct? Right. -Yonik

Re: Percolate feature?

2013-08-09 Thread Yonik Seeley
*All* of the terms in the field must be matched by the querynot vice-versa. And no, we don't have a query for that out of the box. To implement, it seems like it would require the total number of terms indexed for a field (for each document). I guess you could also index start and end tokens

Re: JSON Update create different copies of the same document

2013-08-08 Thread Yonik Seeley
On Thu, Aug 8, 2013 at 4:58 AM, Bruno René Santos brunor...@gmail.com wrote: I thought that by adding a new document with the same id on Solr the document already on Solr would be updated with the new info. Yes, this should be the case. But both documents appear on the search results... How

Re: How to uncache a query to debug?

2013-08-03 Thread Yonik Seeley
On Fri, Aug 2, 2013 at 7:34 AM, Erick Erickson erickerick...@gmail.com wrote: My guess is you're hitting the first time something is done, like sorting or some such occasionally on these queries by chance when a new searcher is opened. And actually, if it was something like sorting (populating

Re: queryResultCache showing all zeros

2013-07-31 Thread Yonik Seeley
On Wed, Jul 31, 2013 at 3:49 PM, Chris Hostetter hossman_luc...@fucit.org wrote: there is however a group.cache.percent option tha you might look into -- but i honestly have no idea if that toggles the use of queryResultCache or something else, i havn't played with it before... That's only a

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

2013-07-27 Thread Yonik Seeley
On Sat, Jul 27, 2013 at 4:30 PM, Roman Chyla roman.ch...@gmail.com wrote: Let me just explain better what I found when I dug inside solr: documents (results of the query) are loaded before they are passed into a writer - so the writers are expecting to encounter the solr documents, but these

Re: paging vs streaming. spawn from (Processing a lot of results in Solr)

2013-07-27 Thread Yonik Seeley
On Sat, Jul 27, 2013 at 5:05 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: anyway, even if writer pulls docs one by one, it doesn't allow to stream a billion of them. Solr writes out DocList, which is really problematic even in deep-paging scenarios. Which part is problematic... the

Re: SolrCloud 4.3.1 - Failure to open existing log file (non fatal) errors under high load

2013-07-25 Thread Yonik Seeley
On Thu, Jul 25, 2013 at 7:44 PM, Tim Vaillancourt t...@elementspace.com wrote: ERROR [2013-07-25 19:34:24.264] [org.apache.solr.common.SolrException] Failure to open existing log file (non fatal) That itself isn't necessarily a problem (and why it says non fatal) - it just means that most

Re: short-circuit OR operator in lucene/solr

2013-07-22 Thread Yonik Seeley
function queries to the rescue! q={!func}def(query($a),query($b),query($c)) a=field1:value1 b=field2:value2 c=field3:value3 def or default function returns the value of the first argument that matches. It's named default because it's more commonly used like def(popularity,50) (return the value

Re: Luke's analysis of Trie Dates

2013-07-18 Thread Yonik Seeley
On Thu, Jul 18, 2013 at 12:53 PM, JohnRodey timothydd...@yahoo.com wrote: I have a TrieDateField dynamic field setup in my schema, pretty standard... dynamicField name=*_tdt type=tdate indexed=true stored=false/ fieldType name=tdate class=solr.TrieDateField omitNorms=true

Re: Ability to specify the server where shard splits go?

2013-07-17 Thread Yonik Seeley
On Wed, Jul 17, 2013 at 12:26 PM, Timothy Potter thelabd...@gmail.com wrote: This is not a problem per se, just want to verify that we're not able to specify which server shard splits are created as of 4.3.1? From what I've seen, the new cores for the sub-shards are created on the leader of

Re: more than 1 join on the same query

2013-07-10 Thread Yonik Seeley
Be careful with URL encoding... that may be messing you up depending on how you are trying to submit the query (and the single you were using as AND) fq={!join from=root_id to=id}type:arm AND attr1=left fq={!join from=root_id to=id}type:leg AND attr1=right -Yonik http://lucidworks.com On Wed,

Re: solr postfilter question

2013-07-10 Thread Yonik Seeley
On Wed, Jul 10, 2013 at 6:08 PM, Rohit Harchandani rhar...@gmail.com wrote: Hey, I am trying to create a plugin which makes use of postfilter. I know that the collect function is called for every document matched, but is there a way i can access all the matched documents upto this point before

Re: Is it possible to facet on existence of a field?

2013-07-08 Thread Yonik Seeley
On Mon, Jul 8, 2013 at 8:18 AM, adfel70 adfe...@gmail.com wrote: I have a field that's only indexed in some of the documents. Can I create a boolean facet on this field by its existence? for instance: yes(124) no(479) Note that the fields' value is not facetable because all its values are

Re: Concurrent Modification Exception

2013-07-06 Thread Yonik Seeley
On Fri, Jul 5, 2013 at 10:57 AM, adityab aditya_ba...@yahoo.com wrote: well our observation leads us that this happens only during spell check. If we turn off the spell check we don't see this issue occurring at all from our 24hrs test run. Are you using any custom components / plugins, or is

Re: Solr 4.x union of cross-joins

2013-07-06 Thread Yonik Seeley
On Sat, Jul 6, 2013 at 2:22 PM, mihaela olteanu mihaela...@yahoo.com wrote: Hello, I have 3 indices that form a hierarchy. Basically these were constructed from 3 tables: parent, child1 and child2 and between parent and children there is a one to many relationship. parent (id,name)

Re: Joins with SolrCloud

2013-07-04 Thread Yonik Seeley
Yes, joins support distributed search fine, provided that the individual documents that are joined reside on the same shard. For example, if you are modeling blogs and posts (one blog object as many posts) shard1 -- joe!blog_info joe!post1 shard2 --

Re: Total Term Frequency per ResultSet in Solr 4.3 ?

2013-07-04 Thread Yonik Seeley
If you just want to retrieve those counts, this seems like simple faceting. q=something facet=true facet.query=product:hunger facet.query=product:games -Yonik http://lucidworks.com On Thu, Jul 4, 2013 at 9:45 AM, Tony Mullins tonymullins...@gmail.com wrote: Hi , I have lots of crawled data,

Re: Total Term Frequency per ResultSet in Solr 4.3 ?

2013-07-04 Thread Yonik Seeley
-frequency in resultset without any modification to Solr source code ? Thanks, Tony On Thu, Jul 4, 2013 at 7:05 PM, Yonik Seeley yo...@lucidworks.com wrote: If you just want to retrieve those counts, this seems like simple faceting. q=something facet=true facet.query=product:hunger

Re: Joins with SolrCloud

2013-07-03 Thread Yonik Seeley
On Wed, Jul 3, 2013 at 5:40 PM, slevytam developm...@the10thfloor.com wrote: Hi Yonik, Can you offer any insight as to how one might ensure that documents reside on the same shard as the document you'd like them to join. For example: I'd like to do a simple join of user actions to a

Re: are fields stored or unstored by default xml

2013-07-01 Thread Yonik Seeley
On Mon, Jul 1, 2013 at 3:50 PM, Jack Krupansky j...@basetechnology.com wrote: stored and indexed both default to true. This is legal: field name=alpha type=string / Actually, for fields I believe the defaults come from the fieldType. The fieldType defaults to true for both indexed and

Re: Joins with SolrCloud

2013-06-28 Thread Yonik Seeley
On Tue, Jun 25, 2013 at 7:55 PM, Upayavira u...@odoko.co.uk wrote: However, if from your example, innerCollection was replicated across all nodes, I would think that should work, because all that comes back from each server when a distributed search happens is the best 'n' matches, so exactly

Re: How to get values of external file field(s) in Solr query?

2013-06-26 Thread Yonik Seeley
On Wed, Jun 26, 2013 at 4:02 PM, Arun Rangarajan arunrangara...@gmail.com wrote: http://docs.lucidworks.com/display/solr/Working+with+External+Files+and+Processes says this about external file fields: They can be used only for function queries or display. I understand how to use them in

Re: shardkey

2013-06-21 Thread Yonik Seeley
On Fri, Jun 21, 2013 at 6:08 PM, Joshi, Shital shital.jo...@gs.com wrote: But now Solr stores composite id in the document id Correct, it's the document id itself that contains everything needed for tje compositeId router to determine the hash. It would only use it to calculate hash key but

Re: Adding pdf/word file using JSON/XML

2013-06-16 Thread Yonik Seeley
On Sun, Jun 16, 2013 at 6:05 PM, Jack Krupansky j...@basetechnology.com wrote: Except, that Solr's divergence from a true, pure REST API is certainly one of the elements of its badness. Most complex systems seem to feel the need to diverge from pure REST for the sake of being practical. From

Solr developer IRC channel

2013-06-10 Thread Yonik Seeley
FYI, I've created a #solr-dev IRC channel for those who contribute to Solr's development. There used to be more of a community feel on some of the IRC channels that's since been lost, so I'm trying to get some of that back with a smaller subset of people interested in developing Solr. The channel

Re: Solr developer IRC channel

2013-06-10 Thread Yonik Seeley
On Mon, Jun 10, 2013 at 5:32 PM, Otis Gospodnetic otis.gospodne...@gmail.com wrote: Mucho good! +1 Why unlogged though? Just curious. Personal preference give it a more informal / slightly more private feel. Some people don't want casual watercooler chat recorded publicized forever.

Re: Clear cache used by Solr

2013-06-07 Thread Yonik Seeley
On Fri, Jun 7, 2013 at 7:32 AM, Erick Erickson erickerick...@gmail.com wrote: I really question whether this is valuable. Much of Solr performance is there explicitly because of caches Right, and it's also the case that certain solr features are coded with the cache in mind (i.e. they will be

Re: facet.missing=true returns null records with zero count also

2013-06-05 Thread Yonik Seeley
On Wed, Jun 5, 2013 at 6:11 PM, Chris Hostetter hossman_luc...@fucit.org wrote: and think that conceptually it doesn't make sense for facet.missing to consider facet.mincount. +1 facet.missing asks for the missing count - regardless of what it is. Although it might make sense in some use cases

Re: Getting tons of EofException with jetty/SolrCloud

2013-05-31 Thread Yonik Seeley
On Fri, May 31, 2013 at 4:11 PM, Jason Hellman jhell...@innoventsolutions.com wrote: Those are default, though autoSoftCommit is commented out by default. Keep in mind about the hard commit running every 15 seconds: it is not updating your searchable data (due to the openSearcher=false

<    1   2   3   4   5   6   7   8   9   10   >