Re: Native library of plugin is loaded for every core

2015-05-27 Thread Alan Woodward
Does it work if you load it via the solr home /lib directory, rather than from the /lib directory of each individual core? Alan Woodward www.flax.co.uk On 27 May 2015, at 08:45, adfel70 wrote: Hi guys, need your help: I added a custom plugins to Solr, to support my applicative needs (one

Re: Removing characters like '\n \n' from indexing

2015-05-27 Thread Erik Hatcher
Edwin - There’s a bunch of built-in update processors you can use, including a script one that allows you to code it dynamically in JavaScript (or other JVM scripting language). See https://cwiki.apache.org/confluence/display/solr/Update+Request+Processors

sort on fields that are not mandatory in each document

2015-05-27 Thread Derek Poh
Hi I am trying to sort on multiple fields. These fields donot necessary exist in every document. sort=sppddrank asc, ddrank asc From the sorted result, it seems that documents which donot have sppddrank field is at the top. How can I make the documents that have the sppddrank field to be

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Rajani Maski
Hi Derek, They are at the fieldType Level. You might find some reference examples in schema.xml using them. https://cwiki.apache.org/confluence/display/solr/Field+Type+Definitions+and+Properties On Wed, May 27, 2015 at 2:30 PM, Derek Poh d...@globalsources.com wrote: Hi Ahmet The

Solr 5 Tutorial which does NOT use dist examples

2015-05-27 Thread Dean Ericson
I am looking for a good tutorial on Solr 5 that does not use the provided dist examples. I am trying to setup a 'test' environment - using embedded zookeeper - that would mimic what would need to be done in test/prod without having to spin up multiple machines to do so. Specifically, we are

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Ahmet Arslan
Hi, I think you are looking for sortMissing* attributes: sortMissingLast and sortMissingFirst attributes are optional attributes are currently supported on types that are sorted internally as strings and on numeric types. Ahmet On Wednesday, May 27, 2015 11:36 AM, Derek Poh

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Derek Poh
Hi Ahmet The sortMissingLast and sortMissingFirst attributes are defined at the field or fieldType level? field name=P_TSRank type=int indexed=true stored=true multiValued=false/ fieldType name=int class=solr.TrieIntField precisionStep=0 positionIncrementGap=0/ On 5/27/2015 4:43 PM,

Re: Displaying a date field that is empty

2015-05-27 Thread Ahmet Arslan
Hi, One approach if to use default value for that field. Missing values represented by that constant value. Other approach is to use a query to capture documents that do not have that field: +*:* -field:[* TO *] I think, newly added exists(field) function query can be used too. Ahmet On

Re: Displaying a date field that is empty

2015-05-27 Thread Alessandro Benedetti
Mmmm if you add a default value, you are going to index a fake date for the null one ( like a specific date you decide). Of course this default value should be different from any other possible date in your system and must not be misleading for users. The reason you want the date to be in the

Re: Native library of plugin is loaded for every core

2015-05-27 Thread Upayavira
In your solr home directory, create a lib directory, and put your jar there. Then you wont have to declare it in solrconfig.xml. That's what Alan is suggesting. Upayavira On Wed, May 27, 2015, at 09:39 AM, adfel70 wrote: Hi Alan, thanks for the reply. I am not sure what did you mean. Currently

Displaying a date field that is empty

2015-05-27 Thread Zheng Lin Edwin Yeo
Hi, Would like to check, is there any method to index date that are empty, and allow it to be searchable and be displayed as an empty field? Currently, for my csv records, there's some records which has no date in the date field, while others have. For those records with no dates, when I search

Re: Native library of plugin is loaded for every core

2015-05-27 Thread adfel70
Hi Alan, thanks for the reply. I am not sure what did you mean. Currently it is loaded from solrconfig.xml lib dir =/path_to_plug_dir/, regex=*.jar / Is there any other way? -- View this message in context:

Re: Index optimize runs in background.

2015-05-27 Thread Upayavira
In this case, optimising makes sense, once the index is generated, you are not updating It. Upayavira On Wed, May 27, 2015, at 06:14 AM, Modassar Ather wrote: Our index has almost 100M documents running on SolrCloud of 5 shards and each shard has an index size of about 170+GB (for the record,

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Alessandro Benedetti
Actually it is both field level and field type level. You decide based on your use case ( can happen that for the same field type , you want sortMissingFirst for one field, and sortMissingLast for another) . I want to add a bonus note, related the ( empty ) and null concept. Be very careful you

Re: Solr 5 Tutorial which does NOT use dist examples

2015-05-27 Thread Alexandre Rafalovitch
Why exactly do you not want to use bundled examples. There is an example that spins several servers next to each other on one machine. Then you can examine everything in there. I am missing something I guess. Regards, Alex. Solr Analyzers, Tokenizers, Filters, URPs and even a newsletter:

Re: Solr 5 Tutorial which does NOT use dist examples

2015-05-27 Thread Dean Ericson
Alex, I already have gone over all the great examples, esp the cloud. My main goal is to stand up my own collections, my own nodes, etc without relying on the cloud example's auto-magic configuration. I am trying to gain insight into where each configuration item needs to go, how to set up a

Re: Solr 5 Tutorial which does NOT use dist examples

2015-05-27 Thread Alexandre Rafalovitch
Well, the cloud initialization steps are in the scripts that set those examples up. Based on your parameters too. To connect with Oracle to read data, you are probably looking for DIH (DataImportHandler) and Oracle as your keywords - not really connected to the scripts. But if you really don't

Dynamic range on numbers

2015-05-27 Thread John Blythe
hi all, i'm attempting to suggest products across a range to users based on dimensions. if there is a 5x10mm Drill Set for instance and a competitor sales something similar enough then i'd like to have it shown. the range, however, would need to be dynamic. i'm thinking for our initial testing

Re: HW requirements

2015-05-27 Thread Toke Eskildsen
Sznajder ForMailingList bs4mailingl...@gmail.com wrote: Could you give me some hints wrt HW requirements for Solr if I need to index about 400 Gigas of text? No. You are providing far too few data for us to guess. 400GB can be handled on a laptop or require 3 strong servers, depending on what

Re: Clarification on Collections API for 5.x

2015-05-27 Thread Erick Erickson
Are you defining shard and replicas here? Or is this just a single-node collection? In any case, this seems unnecessary. You'd get the same thing by having your uploading the config set to ZK, then just issuing a Collections CREATE command, specifying the node to use if desired. What you're doing

Re: Dynamic range on numbers

2015-05-27 Thread John Blythe
thanks erick. will give it a whirl later today and report back tonight or tomorrow. i imagine i'll have some more questions crop up :) best, -- *John Blythe* Product Manager Lead Developer 251.605.3071 | j...@curvolabs.com www.curvolabs.com 58 Adams Ave Evansville, IN 47713 On Wed, May 27,

Re: Start Scripts

2015-05-27 Thread Erick Erickson
Hmmm, this is a little confused I think. bq: copies all necessary config file accordingly This is handled for you, with the caveat that you have to push the configs to Zookeeper as a separate step with the zk client. Once that's done, just use the Collections API to create the collection. See

Copying index from one Solr cloud to other Solr cloud

2015-05-27 Thread Manohar Sripada
I am using Solr cloud 4.7.2. We have around 100 collections spread across 16 Solr nodes. Also, there are 5 dedicated servers for running Zookeeper. I want to move all these collections data (or collections) to a completely different solr cloud. How to achieve this? The Zookeeper servers for this

distributed search limitations via SolrCloud

2015-05-27 Thread Vishal Swaroop
Currently, we have SOLR configured on single linux server (24 GB physical memory) with multiple cores. We are using SOLR joins (https://wiki.apache.org/solr/Join) across cores on this single server. But, as data will grow to ~2 billion we need to assess whether we’ll need to run SolrCloud as In a

Re: distributed search limitations via SolrCloud

2015-05-27 Thread Erick Erickson
Hard to say. I've seen 20M doc be the place you need to consider sharding/SolrCloud. I've seen 300M docs be the place you need to start sharding. That said I'm quite sure you'll need to shard before you get to 2B. There's no good reason to delay that process. You'll have to do something about the

Ability to load solrcore.properties from zookeeper

2015-05-27 Thread Steve Davids
I am attempting to override some properties in my solrconfig.xml file by specifying properties in a solrcore.properties file which is uploaded in Zookeeper's collections/conf directory, though when I go to create a new collection those properties are never loaded. One work-around is to specify

Re: Clarification on Collections API for 5.x

2015-05-27 Thread Jim . Musil
bump On 5/21/15, 9:06 AM, Jim.Musil jim.mu...@target.com wrote: Hi, In the guide for moving from Solr 4.x to 5.x, it states the following: Solr 5.0 only supports creating and removing SolrCloud collections through the Collections

HW requirements

2015-05-27 Thread Sznajder ForMailingList
Hi , Could you give me some hints wrt HW requirements for Solr if I need to index about 400 Gigas of text? Thanks Benjamin

Re: Index optimize runs in background.

2015-05-27 Thread Erick Erickson
All strange of course. What do your Solr logs show when this happens? And how reproducible is this? Best, Erick On Wed, May 27, 2015 at 4:00 AM, Upayavira u...@odoko.co.uk wrote: In this case, optimising makes sense, once the index is generated, you are not updating It. Upayavira On Wed,

Re: Dynamic range on numbers

2015-05-27 Thread Erick Erickson
1 tfloat 2 fq=dimField:[4.5 TO 5.5] or even use frange to set the lower and upper bounds via function Best, Erick On Wed, May 27, 2015 at 5:29 AM, John Blythe j...@curvolabs.com wrote: hi all, i'm attempting to suggest products across a range to users based on dimensions. if there is a

Start Scripts

2015-05-27 Thread Dean Ericson
Does anyone have any simple SOLR startup scripts that they're willing to share? I am looking for a script that creates a custom collection and copies all necessary config file accordingly. Thanks in advance, Dean

Re: Removing characters like '\n \n' from indexing

2015-05-27 Thread Erick Erickson
The other alternative is to use SolrJ to parse the documents and do your processing there. Here's an article on the pros/cons and an example program. https://lucidworks.com/blog/indexing-with-solrj/ Best, Erick On Wed, May 27, 2015 at 1:57 AM, Erik Hatcher erik.hatc...@gmail.com wrote: Edwin -

Re: Solr 5 Tutorial which does NOT use dist examples

2015-05-27 Thread Erick Erickson
bin/solr start -d techproducts that will set up a non-cloud (i.e. stand-alone) Solr that you can use as a model. Note that Solr uses core discovery, which basically recursively descends from SOLR_HOME and any time it finds a core.properties file it assumes it's at the root of another core and

Re: distributed search limitations via SolrCloud

2015-05-27 Thread Vishal Swaroop
Thanks a lot Erick... You are right we should not delay moving to sharding/SolrCloud process. As you all are expert... currently we are using SOLR 4.7.. Do you suggest we should move to latest SOLR release 5.1.0 ? or we can manage the above issue using SOLR 4.7 Regards Vishal On Wed, May 27,

Re: Start Scripts

2015-05-27 Thread Upayavira
I wonder if, Dean, you are using an older. Take a look in the bin/ directory of any newer Solr, preferably 5.x and you'll see quite substantial start scripts. Upayavira On Wed, May 27, 2015, at 07:11 PM, Erick Erickson wrote: Hmmm, this is a little confused I think. bq: copies all necessary

Re: Clarification on Collections API for 5.x

2015-05-27 Thread Jim . Musil
Thanks for the clarification! On 5/27/15, 12:00 PM, Erick Erickson erickerick...@gmail.com wrote: Are you defining shard and replicas here? Or is this just a single-node collection? In any case, this seems unnecessary. You'd get the same thing by having your uploading the config set to ZK, then

Re: distributed search limitations via SolrCloud

2015-05-27 Thread Erick Erickson
I'd move to Solr 4.10.3 at least, but preferably Solr 5.x. Solr 5.2 is being readied for release as we speak, it'll probably be available in a week or so barring unforeseen problems and that's the one I'd go with by preference. Do be aware, though, that the 5.x Solr world deprecates using a war

Re: Removing characters like '\n \n' from indexing

2015-05-27 Thread Shawn Heisey
On 5/26/2015 10:16 PM, Zheng Lin Edwin Yeo wrote: I tried to follow the example here https://wiki.apache.org/solr/UpdateRequestProcessor, by putting the updateRequestProcessorChain in my solrconfig.xml But I'm getting the following error when I tried to reload the core. Caused by:

Re: Difference in running Solr with Jetty internally or externally

2015-05-27 Thread Zheng Lin Edwin Yeo
Thank you Erick! I'll use the internal Jetty and start Solr with the bin/solr start scripts. Regards, Edwin On 26 May 2015 at 12:20, Erick Erickson erickerick...@gmail.com wrote: Actually, just use the new bin/solr start scripts and ignore whether it's running Jetty under the covers or not

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Derek Poh
Got it. Thank you Rajani. On 5/27/2015 5:34 PM, Rajani Maski wrote: Hi Derek, They are at the fieldType Level. You might find some reference examples in schema.xml using them. https://cwiki.apache.org/confluence/display/solr/Field+Type+Definitions+and+Properties On Wed, May 27, 2015 at 2:30

SolrCloud: Creating more shard at runtime will lower down the load?

2015-05-27 Thread Aman Tandon
Hi, I have a question regarding the solr cloud. The load on our search server are increasing day by day as our no of visitors are keep on increasing. So I have a scenario, I want to slice the data at the Runtime, by creating the more shards of the data. *i)* Does it affect the current queries

Re: Ability to load solrcore.properties from zookeeper

2015-05-27 Thread Chris Hostetter
: I am attempting to override some properties in my solrconfig.xml file by : specifying properties in a solrcore.properties file which is uploaded in : Zookeeper's collections/conf directory, though when I go to create a new : collection those properties are never loaded. One work-around is to

Re: sort on fields that are not mandatory in each document

2015-05-27 Thread Derek Poh
Oh ok. Thank youAlessandro. On 5/27/2015 6:07 PM, Alessandro Benedetti wrote: Actually it is both field level and field type level. You decide based on your use case ( can happen that for the same field type , you want sortMissingFirst for one field, and sortMissingLast for another) . I want

Re: docValues: Can we apply synonym

2015-05-27 Thread Aman Tandon
Ok and what synonym processor you is talking about maybe it could help ? With Regards Aman Tandon On Thu, May 28, 2015 at 4:01 AM, Reitzel, Charles charles.reit...@tiaa-cref.org wrote: Sorry, my bad. The synonym processor I mention works differently. It's an extension of the EDisMax query

RE: docValues: Can we apply synonym

2015-05-27 Thread Reitzel, Charles
Is there any reason you cannot apply the synonyms at query time? Applying synonyms at indexing time has problems, e.g. polluting the term frequency for synonyms added, preventing distance queries, ... Since city names often have multiple terms, e.g. New York, Den Hague, etc., I would

RE: When is too many fields in qf is too many?

2015-05-27 Thread Reitzel, Charles
One request handler per view? I think if you are able to make the actual view in use for the current request a single value (vs. all views that the user could use over time), it would keep the qf list down to a manageable size (e.g. specified within the request handler XML). Not sure if

RE: docValues: Can we apply synonym

2015-05-27 Thread Reitzel, Charles
But the query analysis isn't on a specific field, it is applied to the query string. -Original Message- From: Aman Tandon [mailto:amantandon...@gmail.com] Sent: Wednesday, May 27, 2015 6:08 PM To: solr-user@lucene.apache.org Subject: Re: docValues: Can we apply synonym Hi Charles, The

Re: docValues: Can we apply synonym

2015-05-27 Thread Aman Tandon
Hi Charles, The problem here is that the docValues works only with primitives data type only like String, int, etc So how could we apply synonym on primitive data type. With Regards Aman Tandon On Thu, May 28, 2015 at 3:19 AM, Reitzel, Charles charles.reit...@tiaa-cref.org wrote: Is there

RE: docValues: Can we apply synonym

2015-05-27 Thread Reitzel, Charles
Sorry, my bad. The synonym processor I mention works differently. It's an extension of the EDisMax query processor and doesn't require field level synonym configs. -Original Message- From: Reitzel, Charles [mailto:charles.reit...@tiaa-cref.org] Sent: Wednesday, May 27, 2015 6:12 PM

Re: Removing characters like '\n \n' from indexing

2015-05-27 Thread Zheng Lin Edwin Yeo
Hi Shawn, Thanks for your reply. So that means the only way for me is to write my own custom class in order for the removing characters like '\n' to work? Regards, Edwin On 27 May 2015 at 14:46, Shawn Heisey apa...@elyograg.org wrote: On 5/26/2015 10:16 PM, Zheng Lin Edwin Yeo wrote: I

Native library of plugin is loaded for every core

2015-05-27 Thread adfel70
Hi guys, need your help: I added a custom plugins to Solr, to support my applicative needs (one index handler and 2 search components), all of them access a native library using JNI. The native library wrapper class loads the library using the regular pattern: public class YWrapper{