FilterCache size should reduce as index grows?

2017-10-04 Thread S G
Hi, Here is a discussion we had recently with a fellow Solr user. It seems reasonable to me and wanted to see if this is an accepted theory. The bit-vectors in filterCache are as long as the maximum number of documents in a core. If there are a billion docs per core, every bit vector will have a

Re: How to Index JSON field Solr 5.3.2

2017-10-03 Thread Deeksha Sharma
> Sent: Tuesday, October 3, 2017 12:58:57 AM To: solr-user@lucene.apache.org Subject: Re: How to Index JSON field Solr 5.3.2 Hi Sharma, I guess you are looking for nested documents: https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-Ne

Re: Keeping the index naturally ordered by some field

2017-10-02 Thread Erick Erickson
ends on > input sorted by a certain value. In this scenario, regular solr sorting is > insufficient as it's performed in post-search, and only collects needed rows > to satisfy the query. The alternative for naturally sorted index is to sort > all the docs myself, and I wish to avoid this.

Re: Keeping the index naturally ordered by some field

2017-10-02 Thread alexpusch
is insufficient as it's performed in post-search, and only collects needed rows to satisfy the query. The alternative for naturally sorted index is to sort all the docs myself, and I wish to avoid this. I use docValues extensively, it really is a great help. Erick, I've tried using

Re: How to Index JSON field Solr 5.3.2

2017-10-02 Thread Emir Arnautović
Hi Sharma, I guess you are looking for nested documents: https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.html#UploadingDatawithIndexHandlers-NestedChildDocuments <https://lucene.apache.org/solr/guide/6_6/uploading-data-with-index-handlers.h

How to Index JSON field Solr 5.3.2

2017-10-02 Thread Deeksha Sharma
Hi everyone, I have created a core and index data in Solr using dataImportHandler. The schema for the core looks like this: This is my data in mysql database: md5:"376463475574058bba96395bfb87" rules: {"fileRules":[{"file_id":1321241,"md5

Re: Keeping the index naturally ordered by some field

2017-10-02 Thread alessandro.benedetti
Hi Alex, just to explore a bit your question, why do you need that ? Do you need to reduce query time ? Have you tried enabling docValues for the fields of interest ? Doc Values seem to me a pretty useful data structure when sorting is a requirement. I am curious to understand why that was not an

Re: Keeping the index naturally ordered by some field

2017-10-01 Thread Erick Erickson
ober 1, 2017, 10:22:45 AM GMT+3, alexpusch <a...@getjaco.com> > wrote: > > > > > > Hello, > We've got a pretty big index (~1B small docs). I'm interested in managing > the index so that the search results would be naturally sorted by a certain > numeric field, without

Re: Keeping the index naturally ordered by some field

2017-10-01 Thread Ahmet Arslan
Hello, We've got a pretty big index (~1B small docs). I'm interested in managing the index so that the search results would be naturally sorted by a certain numeric field, without specifying the actual sort field in query time. My first attempt was using SortingMergePolicyFactory. I've

Keeping the index naturally ordered by some field

2017-10-01 Thread alexpusch
Hello, We've got a pretty big index (~1B small docs). I'm interested in managing the index so that the search results would be naturally sorted by a certain numeric field, without specifying the actual sort field in query time. My first attempt was using SortingMergePolicyFactory. I've found

Re: Solr 7.0.0 -- can it use a 6.5.0 data repository (index)

2017-09-27 Thread Cassandra Targett
gt; > -Original Message- > From: Stefan Matheis [mailto:matheis.ste...@gmail.com] > Sent: Wednesday, September 27, 2017 12:32 PM > To: solr-user@lucene.apache.org > Subject: Re: Solr 7.0.0 -- can it use a 6.5.0 data repository (index) > > That sounds like > https://

RE: Solr 7.0.0 -- can it use a 6.5.0 data repository (index)

2017-09-27 Thread Wayne L. Johnson
PM To: solr-user@lucene.apache.org Subject: Re: Solr 7.0.0 -- can it use a 6.5.0 data repository (index) That sounds like https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_SOLR-2D11406=DwIFaQ=z0adcvxXWKG6LAMN6dVEqQ=4gLDKHTqOXldY2aQti2VNXYWPtqa1bUKE6MA9VrIJfU

Re: Solr 7.0.0 -- can it use a 6.5.0 data repository (index)

2017-09-27 Thread Stefan Matheis
That sounds like https://issues.apache.org/jira/browse/SOLR-11406 if i'm not mistaken? -Stefan On Sep 27, 2017 8:20 PM, "Wayne L. Johnson" <wjohn...@familysearch.org> wrote: > I’m testing Solr 7.0.0. When I start with an empty index, Solr comes up > just fine, I can a

Solr 7.0.0 -- can it use a 6.5.0 data repository (index)

2017-09-27 Thread Wayne L. Johnson
I'm testing Solr 7.0.0. When I start with an empty index, Solr comes up just fine, I can add documents and query documents. However when I start with an already-populated set of documents (from 6.5.0), Solr will not start. The relevant portion of the traceback seems to be: Caused

Re: Solr Spatial Index and Data

2017-09-17 Thread Furkan KAMACI
any. Kind Regards, Furkan KAMACI On Thu, Sep 14, 2017 at 1:34 PM, Rick Leir <rl...@leirtech.com> wrote: > hi Can Ezgi > > First of all, i want to use spatial index for my data include polyghons > and points. But solr indexed first 18 rows, other rows not indexed. > > Do a

Re: Adding UniqueKey to an existing Solr 6.4 Index

2017-09-15 Thread Erick Erickson
Not really. Do note that atomic updates require 1> all _original_ fields (i.e. fields that are _not_ destinations for copyFields) have stored=true 2> no destination of a copyField has stored=true 3> compose the original document from stored fields and re-index the doc. This latter j

Adding UniqueKey to an existing Solr 6.4 Index

2017-09-15 Thread Pankaj Gurumukhi
Hello, I have a single node Solr 6.4 server, with a Index of 100 Million documents. The default "id" is the primary key of this index. Now, I would like to setup an update process to insert new documents, and update existing documents based on availability of value in another

Re: Solr Spatial Index and Data

2017-09-14 Thread Rick Leir
hi Can Ezgi > First of all, i want to use spatial index for my data include polyghons and points. But solr indexed first 18 rows, other rows not indexed. Do all rows have a unique id field? Are there errors in the logfile? cheers -- Rick .

Solr Spatial Index and Data

2017-09-14 Thread Can Ezgi Aydemir
Hi everyone, First of all, i want to use spatial index for my data include polyghons and points. But solr indexed first 18 rows, other rows not indexed. I need sample datas include polyghons and points. Other problem, i will write spatial query this datas. This spatial query include

Re: Freeze Index

2017-09-14 Thread Toke Eskildsen
On Wed, 2017-09-13 at 11:56 -0700, fabigol wrote: > my problem is that my index freeze several time and i don't know why. > So i lost all the data of my index. > I have 14 million of documents from postgresql database. I have an > only node with 31 GO for my JVM and my server has 64

Re: Freeze Index

2017-09-13 Thread Rick Leir
Fabien, What do you see in the logfile at the time of the freeze? Cheers -- Rick On September 13, 2017 3:01:17 PM EDT, fabigol <fabien.stou...@vialtis.com> wrote: >hi, >my problem is that my index freeze several time and i don't know why. >So i >lost all the data of my index.

Freeze Index

2017-09-13 Thread fabigol
hi, my problem is that my index freeze several time and i don't know why. So i lost all the data of my index. I have 14 million of documents from postgresql database. I have an only node with 31 GO for my JVM and my server has 64GO. My index make 6 GO on the HDD. Is it a good configuration

Freeze Index

2017-09-13 Thread fabigol
hi, my problem is that my index freeze several time and i don't know why. So i lost all the data of my index. I have 14 million of documents from postgresql database. I have an only node with 31 GO for my JVM and my server has 64GO. My index make 6 GO on the HDD. Is it a good configuration

Re: Index relational database

2017-08-31 Thread Erick Erickson
t;>> >>> wunder >>> Walter Underwood >>> wun...@wunderwood.org >>> http://observer.wunderwood.org/ (my blog) >>> >>> >>>> On Aug 30, 2017, at 9:14 AM, Erick Erickson <erickerick...@gmail.com> >>> wrote: >>>> >

Re: Index relational database

2017-08-31 Thread Walter Underwood
Erick Erickson <erickerick...@gmail.com> >> wrote: >>> >>> First, it's often best, by far, to denormalize the data in your solr >> index, >>> that's what I'd explore first. >>> >>> If you can't do that, the join query parser might work fo

Re: Solr index getting replaced instead of merged

2017-08-31 Thread David Hastings
al, Harshal (GE Digital) > Sent: Wednesday, August 30, 2017 4:36 PM > To: 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org> > Cc: Singh, Susnata (GE Digital) <susnata.si...@ge.com> > Subject: Solr index getting replaced instead of merged > > Hello Guys, > >

RE: Solr index getting replaced instead of merged

2017-08-31 Thread Agrawal, Harshal (GE Digital)
4:36 PM To: 'solr-user@lucene.apache.org' <solr-user@lucene.apache.org> Cc: Singh, Susnata (GE Digital) <susnata.si...@ge.com> Subject: Solr index getting replaced instead of merged Hello Guys, I have installed solr in my local system and was able to connect to Teradata successfully

Re: Index relational database

2017-08-31 Thread David Hastings
ttp://observer.wunderwood.org/ (my blog) > > > > > > > On Aug 30, 2017, at 9:14 AM, Erick Erickson <erickerick...@gmail.com> > > wrote: > > > > > > First, it's often best, by far, to denormalize the data in your solr > > index, > >

Re: Index relational database

2017-08-31 Thread Renuka Srishti
mented as a view or as SQL, but that is a useful > mental > > > model for people starting from a relational background. > > > > > > wunder > > > Walter Underwood > > > wun...@wunderwood.org > > > http://observer.wunderwood.org/ (my blog) > &g

Re: Index relational database

2017-08-31 Thread Susheel Kumar
.org/ (my blog) > > > > > > > On Aug 30, 2017, at 9:14 AM, Erick Erickson <erickerick...@gmail.com> > > wrote: > > > > > > First, it's often best, by far, to denormalize the data in your solr > > index, > > > that's what I'd explore

Re: Index relational database

2017-08-31 Thread Renuka Srishti
http://observer.wunderwood.org/ (my blog) > > > > On Aug 30, 2017, at 9:14 AM, Erick Erickson <erickerick...@gmail.com> > wrote: > > > > First, it's often best, by far, to denormalize the data in your solr > index, > > that's what I'd explore first. > >

Re: Index relational database

2017-08-30 Thread Walter Underwood
wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Aug 30, 2017, at 9:14 AM, Erick Erickson <erickerick...@gmail.com> wrote: > > First, it's often best, by far, to denormalize the data in your solr index, > that's what I'd explore first. > > If you can't

Re: Index relational database

2017-08-30 Thread Erick Erickson
First, it's often best, by far, to denormalize the data in your solr index, that's what I'd explore first. If you can't do that, the join query parser might work for you. On Aug 30, 2017 4:49 AM, "Renuka Srishti" <renuka.srisht...@gmail.com> wrote: > Thanks Susheel for yo

Re: Solr index getting replaced instead of merged

2017-08-30 Thread Gurdeep Singh
solr in my local system and was able to connect to Teradata > successfully. > For single table I am able to index the data and query it also but when I am > trying for multiple tables in the same schema and doing indexing one by one > respectively. > I can see datasets getting replaced i

Solr index getting replaced instead of merged

2017-08-30 Thread Agrawal, Harshal (GE Digital)
Hello Guys, I have installed solr in my local system and was able to connect to Teradata successfully. For single table I am able to index the data and query it also but when I am trying for multiple tables in the same schema and doing indexing one by one respectively. I can see datasets

Re: Index relational database

2017-08-30 Thread Renuka Srishti
, desc, city > etc.) > b) What is that you want to show part of search result (name, city etc.) > > Based on above two questions, you would know what data to pull in from > relational database and create solr schema and index the data. > > You may first try to denormalize / flat

solr index replace with index from another environment

2017-08-28 Thread Satya Marivada
Hi there, We are using solr-6.3.0 and have the need to replace the solr index in production with the solr index from another environment on periodical basis. But the jvms have to be recycled for the updated index to take effect. Is there any way this can be achieved without restarting the jvms

Re: Index relational database

2017-08-28 Thread Susheel Kumar
questions, you would know what data to pull in from relational database and create solr schema and index the data. You may first try to denormalize / flatten the structure so that you deal with one collection/schema and query upon it. HTH. Thanks, Susheel On Mon, Aug 28, 2017 at 8:04 AM, Renuka

Index relational database

2017-08-28 Thread Renuka Srishti
Hii, What is the best way to index relational database, and how it impacts on the performance? Thanks Renuka Srishti

Re: Correct approach to copy index between solr clouds?

2017-08-26 Thread Erick Erickson
write.lock is used whenever a core(replica) wants to, well, write to the index. Each individual replica is sure to only write to the index with one thread. If two threads were to write to an index, there's a very good chance the index will be corrupt, so it's a safeguard against two or more

Re: Correct approach to copy index between solr clouds?

2017-08-26 Thread Wei
that you don't copy > over the write.lock file however as you may not be able to start > replicas if that's there. > > There's a relatively little-known third option. You an (ab)use the > replication API "fetchindex" command, see: > https://cwiki.apache.org/conf

Re: Correct approach to copy index between solr clouds?

2017-08-26 Thread Erick Erickson
apache.org/confluence/display/solr/Index+Replication to pull the index from Cloud B to replicas on Cloud A. That has the advantage of working even if you are actively indexing to Cloud B. NOTE: currently you cannot _query_ CloudA (the target) while the fetchindex is going on, but I doubt you really care

Correct approach to copy index between solr clouds?

2017-08-25 Thread Wei
behind, we want to bulk copy the binary index from B to A. We have tried two approaches: Approach 1. For cloud A: a. delete collection to wipe out everything b. create new collection (data is empty now) c. shut down solr server d. copy binary index from cloud B

Re: Solr caching the index file make server refuse serving

2017-08-24 Thread Erick Erickson
totle > > Problem: > > When we start solrcloud ,the cached index will make memory 98% or > more used . And if we continue to index document (batch commit 10 000 > documents),one or more server will refuse serving.Cannot login wia ssh,even > refuse the monitor. > > So,how can I limit the solr’s caching index to memory behavior? > > Anyone thanks! >

Solr caching the index file make server refuse serving

2017-08-24 Thread 陈永龙
Hello, ENV: solrcloud 6.3 3*dell server 128G 12cores 4.3T /server 3 solr node /server 20G /node (with parameter �Cm 20G) 10 billlion documents totle Problem: When we start solrcloud ,the cached index will make memory 98% or more used . And if we continue to index document

Re: Move index directory to another partition

2017-08-10 Thread Mahmoud Almokadem
than setting dataDir in core.properties for every core, > especially in a cloud install. > > Agreed. Nothing in what I said precludes this. If you don't specify > dataDir, > then the index for a new replica goes in the default place, i.e. under > your install > directory usuall

Building Solr index from AEM using and ELB

2017-08-09 Thread Wahlgren Peter
I am looking for lessons learned or problems seen when building a Solr index from AEM using a Solr cluster with content passing through an ELB. Our configuration is AEM 6.1 indexing to a cluster of Solr servers running version 4.7.1. When building an index with a smaller data set - 4 million

Re: Move index directory to another partition

2017-08-05 Thread Erick Erickson
be a lot easier than setting dataDir in core.properties for every core, especially in a cloud install. Agreed. Nothing in what I said precludes this. If you don't specify dataDir, then the index for a new replica goes in the default place, i.e. under your install directory usually. In your case under your

Re: Move index directory to another partition

2017-08-04 Thread Shawn Heisey
On 8/2/2017 9:17 AM, Erick Erickson wrote: > Not entirely sure about AWS intricacies, but getting a new replica to > use a particular index directory in the general case is just > specifying dataDir=some_directory on the ADDREPLICA command. The index > just needs an HTTP connection (

Re: mixed index with commongrams

2017-08-03 Thread David Hastings
he > Courts > > of Equity of the United States") > > 2017-08-02 02:16:36 : 54749/1000 secs : ("The American Cause") > > 2017-08-02 19:27:58 : 54561/1000 secs : ("register of the department of > > justice") > > > > which could all be annihi

Re: mixed index with commongrams

2017-08-03 Thread Walter Underwood
secs : ("register of the department of > justice") > > which could all be annihilated with CG's, at the expense, according to HT, > of a 40% increase in index size. > > > > On Thu, Aug 3, 2017 at 11:21 AM, Erick Erickson <erickerick...@gmail.com> > wro

Re: mixed index with commongrams

2017-08-03 Thread David Hastings
rding to HT, of a 40% increase in index size. On Thu, Aug 3, 2017 at 11:21 AM, Erick Erickson <erickerick...@gmail.com> wrote: > bq: will that search still return results form the earlier documents > as well as the new ones > > In a word, "no". By definition the analy

Re: mixed index with commongrams

2017-08-03 Thread Erick Erickson
bq: will that search still return results form the earlier documents as well as the new ones In a word, "no". By definition the analysis chain applied at index time puts tokens in the index and that's all you have to search against for the doc unless and until you re-index the docu

mixed index with commongrams

2017-08-03 Thread David Hastings
Hey all, I have yet to run an experiment to test this but was wondering if anyone knows the answer ahead of time. If i have an index built with documents before implementing the commongrams filter, then enable it, and start adding documents that have the filter/tokenizer applied, will searches

Re: Custom Sort option to apply at SOLR index

2017-08-02 Thread Erick Erickson
CD1234 > ABCD5678 > > *Expected Descending order* > > ABCD5678 > ABCD1234 > 5678ABCD > 1234ABCD > 1234#ABCD > #2345DBCA > #2345ACBD > #2345ABCD > > Thanks & Regards, > Paddy > > > > -- > View this message in context: > http://lucene.472066.n3.nabble.com/Custom-Sort-option-to-apply-at-SOLR-index-tp4348787.html > Sent from the Solr - User mailing list archive at Nabble.com.

Re: Move index directory to another partition

2017-08-02 Thread Erick Erickson
Shawn: Not entirely sure about AWS intricacies, but getting a new replica to use a particular index directory in the general case is just specifying dataDir=some_directory on the ADDREPLICA command. The index just needs an HTTP connection (uses the old replication process) so nothing huge

RE: Solr Index issue on string type while querying

2017-08-02 Thread padmanabhan
Thank you Matt for the reply. my apologize on the clarity about the problem statement. The problem was with the source attribute value defined at the source system. Source system with the heightSquareTube_string_mv: 90 - 100 mm Solr index converts the xml or html code to its symbol

Custom Sort option to apply at SOLR index

2017-08-02 Thread padmanabhan
egards, Paddy -- View this message in context: http://lucene.472066.n3.nabble.com/Custom-Sort-option-to-apply-at-SOLR-index-tp4348787.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Move index directory to another partition

2017-08-01 Thread Dave
To add to this, not sure of solr cloud uses it, but you're going to want to destroy the wrote.lock file as well > On Aug 1, 2017, at 9:31 PM, Shawn Heisey wrote: > >> On 8/1/2017 7:09 PM, Erick Erickson wrote: >> WARNING: what I currently understand about the limitations

Re: Move index directory to another partition

2017-08-01 Thread Shawn Heisey
On 8/1/2017 7:09 PM, Erick Erickson wrote: > WARNING: what I currently understand about the limitations of AWS > could fill volumes so I might be completely out to lunch. > > If you ADDREPLICA with the new replica's data residing on the new EBS > volume, then wait for it to sync (which it'll do

Re: Move index directory to another partition

2017-08-01 Thread Erick Erickson
y I'm using one >> replication factor but I think the downtime will be less than five minutes >> after following your steps. >> >> But how can I start Solr backup or why should I run it although I copied >> the index and changed theo path? >> >> And what do you me

Re: Move index directory to another partition

2017-08-01 Thread Shawn Heisey
n it although I copied > the index and changed theo path? > > And what do you mean with "Using multiple passes with rsync"? The first time you copy the data, which you could do with cp if you want, the time required will be limited by the size of the data and the speed of

Re: Move index directory to another partition

2017-08-01 Thread Mahmoud Almokadem
Thanks Shawn, I'm using ubuntu and I'll try rsync command. Unfortunately I'm using one replication factor but I think the downtime will be less than five minutes after following your steps. But how can I start Solr backup or why should I run it although I copied the index and changed theo path

Re: Move index directory to another partition

2017-08-01 Thread Walter Underwood
hmoud Almokadem wrote: >> I've a SolrCloud of four instances on Amazon and the EBS volumes that >> contain the data on everynode is going to be full, unfortunately Amazon >> doesn't support expanding the EBS. So, I'll attach larger EBS volumes to >> move the index to. >

Re: Move index directory to another partition

2017-08-01 Thread Shawn Heisey
On 7/31/2017 12:28 PM, Mahmoud Almokadem wrote: > I've a SolrCloud of four instances on Amazon and the EBS volumes that > contain the data on everynode is going to be full, unfortunately Amazon > doesn't support expanding the EBS. So, I'll attach larger EBS volumes to > move the ind

Move index directory to another partition

2017-07-31 Thread Mahmoud Almokadem
Hello, I've a SolrCloud of four instances on Amazon and the EBS volumes that contain the data on everynode is going to be full, unfortunately Amazon doesn't support expanding the EBS. So, I'll attach larger EBS volumes to move the index to. I can stop the updates on the index, but I'm afraid

Re: index version - replicable versus searching

2017-07-25 Thread Erick Erickson
Ronald: Actually, people generally don't search on master ;). The idea is that master is configured for heavy indexing and then people search on the slaves which are configured for heavy query loads (e.g. memory, autowarming, whatever may be different). Which is it's own problem since the time

RE: index version - replicable versus searching

2017-07-25 Thread Stanonik, Ronald
Bingo! Right on both counts! opensearcher was false. When I changed it to true, then I could see that master(searching) and master(replicable) both changed. And autocommit.maxtime is causing a commit on the master. Who uses master(replicable)? It seems for my simple master/slave

Re: Lucene index corruption and recovery

2017-07-25 Thread sputul
Another sanity check. With deletion, only option would be to reindex those documents. Could someone please let me know if I am missing anything or if I am on track here. Thanks. -- View this message in context: http://lucene.472066.n3.nabble.com/Lucene-index-corruption-and-recovery

Lucene index corruption and recovery

2017-07-24 Thread Putul S
While trying to upgrade 100G index from Solr 4 to 5, check index (actually updater) indicates that the index is corrupted. Hence, I ran check index to fix the index which showed broken segment warning and then deleted those documents. I then ran index update on the fixed index which upgraded fine

Re: index version - replicable versus searching

2017-07-24 Thread Erick Erickson
has openSearcher=false. This closed all open segments (i.e. the segments with the new docs) 2> the slave replicated the closed segments and opened a new searcher on the index, so it shows the new docs 3> the master still hasn't opened a new searcher so continues to not be able to see the new

index version - replicable versus searching

2017-07-24 Thread Stanonik, Ronald
I'm testing replication on solr 5.5.0. I set up one master and one slave. The index versions match; that is, master(replicable), master(searching), and slave(searching) are the same. I make a change to the index on the master, but do not commit yet. As expected, the version master(replicable

Re: index new discovered fileds of different types

2017-07-10 Thread Jan Høydahl
I think Thaer’s answer clarify how they do it. So at the time they assemble the full Solr doc to index, there may be a new field name not known in advance, but to my understanding the RDF source contains information on the type (else they could not do the mapping to dynamic field either) and so

Re: index new discovered fileds of different types

2017-07-10 Thread Thaer Sammar
es on it are sent to a kafka queue. and we > >have a consumer which listen to the queue and update the Solr index. > > > >regards, > >Thaer > > > >On 7 July 2017 at 10:53, Jan Høydahl <jan@cominvent.com> wrote: > > > >> If you do not need the

Re: index new discovered fileds of different types

2017-07-09 Thread Rick Leir
personally written a Python script to parse RDF files into an in-memory graph structure and then pull data from that structure to index to Solr. I.e. you may perfectly well have RDF (nt, turtle, whatever) as source but index sub structures in very specific ways. Anyway, as Erick points out, that’s

Re: index new discovered fileds of different types

2017-07-09 Thread Jan Høydahl
Hi, I have personally written a Python script to parse RDF files into an in-memory graph structure and then pull data from that structure to index to Solr. I.e. you may perfectly well have RDF (nt, turtle, whatever) as source but index sub structures in very specific ways. Anyway, as Erick

Re: index new discovered fileds of different types

2017-07-07 Thread Rick Leir
fields are known. We get the >data >from RDF database (which changes continuously). To be more specific, we >have a database and all changes on it are sent to a kafka queue. and we >have a consumer which listen to the queue and update the Solr index. > >regards, >Thaer &g

Re: index new discovered fileds of different types

2017-07-07 Thread Erick Erickson
a consumer which listen to the queue and update the Solr index. > > regards, > Thaer > > On 7 July 2017 at 10:53, Jan Høydahl <jan@cominvent.com> wrote: > >> If you do not need the flexibility of dynamic fields, don’t use them. >> Sounds to me that you really

Re: index new discovered fileds of different types

2017-07-07 Thread Thaer Sammar
. and we have a consumer which listen to the queue and update the Solr index. regards, Thaer On 7 July 2017 at 10:53, Jan Høydahl <jan@cominvent.com> wrote: > If you do not need the flexibility of dynamic fields, don’t use them. > Sounds to me that you really want a field “price”

Re: index new discovered fileds of different types

2017-07-07 Thread Jan Høydahl
If you do not need the flexibility of dynamic fields, don’t use them. Sounds to me that you really want a field “price” to be float and a field “birthdate” to be of type date etc. If so, simply create your schema (either manually, through Schema API or using schemaless) up front and index each

Re: index new discovered fileds of different types

2017-07-05 Thread Erick Erickson
e >> >> On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t.sam...@geophy.com> wrote: >> >> > Hi, >> > We are trying to index documents of different types. Document have >> > different fields. fields are known at indexing time. We run a query on a >>

Re: index new discovered fileds of different types

2017-07-05 Thread Thaer Sammar
://cwiki.apache.org/confluence/display/solr/Schemaless+Mode > > On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t.sam...@geophy.com> wrote: > > > Hi, > > We are trying to index documents of different types. Document have > > different fields. fields are known at indexing

Re: index new discovered fileds of different types

2017-07-05 Thread Furkan KAMACI
Hi Thaer, Do you use schemeless mode [1] ? Kind Regards, Furkan KAMACI [1] https://cwiki.apache.org/confluence/display/solr/Schemaless+Mode On Wed, Jul 5, 2017 at 4:23 PM, Thaer Sammar <t.sam...@geophy.com> wrote: > Hi, > We are trying to index documents of different types. D

index new discovered fileds of different types

2017-07-05 Thread Thaer Sammar
Hi, We are trying to index documents of different types. Document have different fields. fields are known at indexing time. We run a query on a database and we index what comes using query variables as field names in solr. Our current solution: we use dynamic fields with prefix, for example

Re: Solr 6.4. Can't index MS Visio vsdx files

2017-07-04 Thread Charlie Hull
l.com] Sent: Tuesday, April 11, 2017 1:56 PM To: solr-user@lucene.apache.org Subject: RE: Solr 6.4. Can't index MS Visio vsdx files Thanks for your responses. Are there any posibilities to ignore parsing errors and continue indexing? because now solr/tika stops parsing whole document if it finds any

RE: Solr 6.4. Can't index MS Visio vsdx files

2017-07-03 Thread Allison, Timothy B.
Sorry. Y, you'll have to update commons-compress to 1.14. -Original Message- From: Gytis Mikuciunas [mailto:gyt...@gmail.com] Sent: Monday, July 3, 2017 9:15 AM To: solr-user@lucene.apache.org Subject: Re: Solr 6.4. Can't index MS Visio vsdx files hi, So I'm back from my long

Re: Solr 6.4. Can't index MS Visio vsdx files

2017-07-03 Thread Gytis Mikuciunas
hi, So I'm back from my long vacations :) I'm trying to bring-up a fresh solr 6.6 standalone instance on windows 2012R2 server. Replaced: poi-*3.15-beta1 ---> poi-*3.16 tika-*1.13 ---> tika-*1.15 Tried to index one txt file and got (with poi and tika files that come out of t

How to index binary files from ftp Servers using Solr DIH?

2017-06-29 Thread Alejandro Rivas Martinez
I need a way to index binary files from ftp servers, using UrlDataSource. I’m doing this locally but I need to do the same from remote sources (Ftp servers). I read a lot and I can’t find any example of indexing binary files from ftps. Is it possible to achieve that? How can I use Data Import

Sharding of index data takes long time.

2017-06-27 Thread chandrushanmugasundaram
I am just trying to shard my index data of size 22GB(1.7M documents) into three shards. The total time for splitting takes about 7 hours. In used the same query that is mentioned in solr collections API. Is there anyway to do that quicker. Can i use REBALANCE API . is that secured

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-27 Thread Joel Bernstein
ks like something wrong/bug in > the > >> > code. Please suggest > >> > > >> > === > >> > let(a=search(collection1, > >> > q=id:9, > >> > fl="id,business_email", > >> > so

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-27 Thread Susheel Kumar
t;> > === >> > let(a=search(collection1, >> > q=id:9, >> > fl="id,business_email", >> > sort="business_email asc"), >> > get(a) >> > ) >> &

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-23 Thread Susheel Kumar
q=id:9, > > fl="id,business_email", > > sort="business_email asc"), > > get(a) > > ) > > > > > > { > > "result-set": { > > "docs": [

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-23 Thread Joel Bernstein
, > q=id:9, > fl="id,business_email", > sort="business_email asc"), > get(a) > ) > > > { > "result-set": { > "docs&qu

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-23 Thread Susheel Kumar
in the code. Please suggest === let(a=search(collection1, q=id:9, fl="id,business_email", sort="business_email asc"), get(a) ) { "result-set": { "docs": [ {

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-23 Thread Joel Bernstein
h EOF:true > assigned to let variable, it gets changed to EXCEPTION "Index 0, Size 0" > etc. > > So let stream not able to handle the stream/results which has only EOF > tuple and breaks the whole let expression block > > > ===Complement inside let > let( &g

Re: Index 0, Size 0 - hashJoin Stream function Error

2017-06-22 Thread Susheel Kumar
Hi Joel, I am able to reproduce this in a simple way. Looks like Let Stream is having some issues. Below complement function works fine if I execute outside let and returns an EOF:true tuple but if a tuple with EOF:true assigned to let variable, it gets changed to EXCEPTION "Index 0, S

Re: Error after moving index

2017-06-22 Thread Erick Erickson
il.com> wrote: > > > > > > > > > BTW, is there a better/recommended way to transfer an > index to another solr? > > > > > > > > > > On Thu, Jun 22, 2017 at 6:09 PM +0200, "Moritz Michael" < > moritz.mu..

Re: Error after moving index

2017-06-22 Thread Susheel Kumar
Usually we index directly into Prod solr than copying from local/lower environments. If that works in your scenario, i would suggest to directly index into Prod than copying/restoring from local Windows env to Linux. On Thu, Jun 22, 2017 at 12:13 PM, Moritz Michael <moritz.mu...@gmail.com>

Re: Error after moving index

2017-06-22 Thread Moritz Michael
BTW, is there a better/recommended way to transfer an index to another solr? On Thu, Jun 22, 2017 at 6:09 PM +0200, "Moritz Michael" <moritz.mu

Re: Error after moving index

2017-06-22 Thread Moritz Michael
_ From: Michael Kuhlmann <k...@solr.info> Sent: Donnerstag, Juni 22, 2017 2:50 PM Subject: Re: Error after moving index To: <solr-user@lucene.apache.org> Hi Moritz, did you stop your local Solr sever before? Copying data fr

<    4   5   6   7   8   9   10   11   12   13   >