Re: Replication in soft commit

2020-09-03 Thread Emir Arnautović
Hi Tushar, This is not usecase suitable for MS model. You should go with Solr Cloud, or if that is an overhead for you, have separate Solr, each doing indexing on its own. Solr provides eventual consistency anyway, so you should have some sort of stickiness in place even if you use MS model.

Re: Replication in soft commit

2020-09-03 Thread Tushar Arora
Hi Emir, Thanks for the response. Actually the use case is real time indexing from DB to solr in every second on the master server using queueing mechanism. So, I think instead of doing hard commits every second we should go for soft commits. And doing hard commits after some intervals. And we

Re: Replication in soft commit

2020-09-03 Thread Emir Arnautović
Hi Tushar, Replication is file based process and hard commit is when segment is flushed to disk. It is not common that you use soft commits on master. The only usecase that I can think of is when you read your index as part of indexing process, but even that is bad practice and should be

Re: Replication of Solr Model and feature store

2020-08-20 Thread Monica Skidmore
t 7, 2020 at 10:28 AM To: "solr-user@lucene.apache.org" , Monica Skidmore Cc: Christine Poerschke Subject: Re: Replication of Solr Model and feature store Hi Monica, Replication is working fine for me. You just have to add the _schema_feature-store.json and _schema_model-s

Re: Replication of Solr Model and feature store

2020-08-07 Thread krishan goyal
https%3A%2F%2Flucene.apache.org%2Fsolr%2Fguide%2F8_6%2Flearning-to-rank.html%23applying-changesdata=01%7C01%7CMonica.Skidmore%40careerbuilder.com%7C65581e5e79414c90832508d82fc8ce21%7C7cc1677566a34e8b80fd5b1f1db15061%7C0sdata=mMqgPhnkjb8h7ETQNaySOBJQ8x%2FP2dtzM%2FgSE1K1FZg%3Dreserved=0 > > From: sol

Re: Replication of Solr Model and feature store

2020-08-06 Thread Monica Skidmore
From: solr-user@lucene.apache.org At: 07/22/20 14:00:59To: solr-user@lucene.apache.org Subject: Re: Replication of Solr Model and feature store Adding more details here I need some help on how to enable the solr LTR model and features on all nodes of a solr cluster.

Re: Replication of Solr Model and feature store

2020-07-28 Thread krishan goyal
solr-user@lucene.apache.org At: 07/22/20 14:00:59To: > solr-user@lucene.apache.org > Subject: Re: Replication of Solr Model and feature store > > Adding more details here > > I need some help on how to enable the solr LTR model and features on all > nodes of a solr clus

Re: Replication of Solr Model and feature store

2020-07-24 Thread Christine Poerschke (BLOOMBERG/ LONDON)
/solr/guide/8_6/learning-to-rank.html#applying-changes From: solr-user@lucene.apache.org At: 07/22/20 14:00:59To: solr-user@lucene.apache.org Subject: Re: Replication of Solr Model and feature store Adding more details here I need some help on how to enable the solr LTR model and features on all

Re: Replication of Solr Model and feature store

2020-07-22 Thread krishan goyal
Adding more details here I need some help on how to enable the solr LTR model and features on all nodes of a solr cluster. I am unable to replicate the model and the feature store though from any master to its slaves with the replication API ? And unable to find any documentation for the same.

Re: Replication of Solr Model and feature store

2020-07-22 Thread krishan goyal
Bump. Any one has an idea how to proceed here ? On Wed, Jul 8, 2020 at 5:41 PM krishan goyal wrote: > Hi, > > How do I enable replication of the model and feature store ? > > Thanks > Krishan >

Re: Replication Iteration

2019-09-13 Thread Paras Lehana
> > Thanks & Regards, > Akreeti Agarwal > > -Original Message- > From: Paras Lehana > Sent: Thursday, September 12, 2019 6:46 PM > To: solr-user@lucene.apache.org > Subject: Re: Replication Iteration > > Hey Akreeti, > > 00:00:10 > > > Have you

RE: Replication Iteration

2019-09-13 Thread Akreeti Agarwal
To: solr-user@lucene.apache.org Subject: Re: Replication Iteration Hey Akreeti, 00:00:10 Have you tried increasing *commitReserveDuration*? Do you have any idea how much time your successful replications take for 2.62 GB? On Wed, 11 Sep 2019 at 22:30, Akreeti Agarwal wrote: > Hi, > &

Re: Replication Iteration

2019-09-12 Thread Paras Lehana
6246 > Swap:0 0 0 > > > Thanks & Regards, > Akreeti Agarwal > > > -Original Message----- > From: Jon Kjær Amundsen > Sent: Wednesday, September 11, 2019 7:28 PM > To: solr-user@lucene.apache.org > Subject: Re: Replication Itera

RE: Replication Iteration

2019-09-11 Thread Akreeti Agarwal
eeti Agarwal -Original Message- From: Jon Kjær Amundsen Sent: Wednesday, September 11, 2019 7:28 PM To: solr-user@lucene.apache.org Subject: Re: Replication Iteration Is it every time it fails, or just sometimes? What is the timestamps on the failed and passed iterations? And how much disk s

Re: Replication Iteration

2019-09-11 Thread Jon Kjær Amundsen
ons. 11. sep. 2019 kl. 15.23 skrev Akreeti Agarwal : > My index size is 2.62 GB, and : > 00:00:10 > > Thanks & Regards, > Akreeti Agarwal > > > -Original Message- > From: Paras Lehana > Sent: Wednesday, September 11, 2019 5:39 PM > To: solr-user@lucen

RE: Replication Iteration

2019-09-11 Thread Akreeti Agarwal
My index size is 2.62 GB, and : 00:00:10 Thanks & Regards, Akreeti Agarwal -Original Message- From: Paras Lehana Sent: Wednesday, September 11, 2019 5:39 PM To: solr-user@lucene.apache.org Subject: Re: Replication Iteration What is the size of your index? Is it too big? How

Re: Replication Iteration

2019-09-11 Thread Paras Lehana
he.solr.common.SolrException: Unable to > download segments_znow completely. Downloaded 0!=2217 > > Thanks & Regards, > Akreeti Agarwal > > -Original Message- > From: Paras Lehana > Sent: Wednesday, September 11, 2019 5:17 PM > To: solr-user@lucene.apache.org > Subje

RE: Replication Iteration

2019-09-11 Thread Akreeti Agarwal
hana Sent: Wednesday, September 11, 2019 5:17 PM To: solr-user@lucene.apache.org Subject: Re: Replication Iteration Hi Akreeti, Have you tried using the old UI to see errors? I had always experienced not seeing status updates about replication in the newer UI. Check for the option on top right of

Re: Replication Iteration

2019-09-11 Thread Paras Lehana
ay, September 11, 2019 12:15 PM > To: solr-user@lucene.apache.org > Subject: Re: Replication Iteration > > It depends on the timestamps. > The red iterations are failed replications and the green are passed > replications. > If the newest timestamp is green the latest replication

RE: Replication Iteration

2019-09-11 Thread Akreeti Agarwal
user@lucene.apache.org Subject: Re: Replication Iteration It depends on the timestamps. The red iterations are failed replications and the green are passed replications. If the newest timestamp is green the latest replication went well, if it is red, it failed. You should check the solr log on the s

Re: Replication Iteration

2019-09-11 Thread Jon Kjær Amundsen
It depends on the timestamps. The red iterations are failed replications and the green are passed replications. If the newest timestamp is green the latest replication went well, if it is red, it failed. You should check the solr log on the slave if a recent replication have failed to see the

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-26 Thread Patrick Bordelon
One other question related to this. I know the change was made for a specific problem that was occurring but has this caused a similar problem as mine with anyone else? We're looking to try changing the second 'if' statement to add an extra conditional to prevent it from performing the

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-25 Thread Patrick Bordelon
I removed the replicate after startup from our solrconfig.xml file. However that didn't solve the issue. When I rebuilt the primary, the associated replicas all went to 0 documents. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-25 Thread Mikhail Khludnev
Ok. probable dropping startup will help. Another idea set replication.enable.master=false and enable it when master index is build after restart. On Tue, Jun 25, 2019 at 6:18 PM Patrick Bordelon < patrick.borde...@coxautoinc.com> wrote: > We are currently using the replicate after commit and

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-25 Thread Patrick Bordelon
We are currently using the replicate after commit and startup ${replication.enable.master:false} commit startup schema.xml,stopwords.txt -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-25 Thread Mikhail Khludnev
Note, it seems like the current Solr's logic relies on persistent master disks. https://github.com/apache/lucene-solr/blob/master/solr/core/src/test/org/apache/solr/handler/TestReplicationHandler.java#L615 On Tue, Jun 25, 2019 at 3:16 PM Mikhail Khludnev wrote: > Hello, Patrick. > Can commit

Re: Replication issue with version 0 index in SOLR 7.5

2019-06-25 Thread Mikhail Khludnev
Hello, Patrick. Can commit help you? On Tue, Jun 25, 2019 at 12:55 AM Patrick Bordelon < patrick.borde...@coxautoinc.com> wrote: > Hi, > > We recently upgraded to SOLR 7.5 in AWS, we had previously been running > SOLR > 6.5. In our current configuration we have our applications broken into a >

Re: Replication error in SOLR-6.5.1

2018-09-26 Thread Erick Erickson
bq. In all my solr servers I have 40% free space Well, clearly that's not enough if you're getting this error: "No space left on device" Solr/Lucene need _at least_ as much free space as the indexes occupy. In some circumstances it can require more. It sounds like you're having an issue with

Re: /replication?command=details does not show infos for all replicas on the core

2018-07-02 Thread Arturas Mazeika
Hi Shawn, hi Erick, hi et al., Very nice clarifications indeed. I also looked at the index replication section. In addition to the clarifications in this thread this brought quite some light into the area (and shows that I need to read solrcloud part of the manual more extensively). Thanks a lot

Re: /replication?command=details does not show infos for all replicas on the core

2018-06-29 Thread Shawn Heisey
On 6/29/2018 8:47 AM, Arturas Mazeika wrote: Out of curiosity: some cores give infos for both shards (through replication query) and some only for one (if you still be able to see the prev post). I wonder why.. Adding to what Erick said: If SolrCloud has initiated a replication on that core

Re: /replication?command=details does not show infos for all replicas on the core

2018-06-29 Thread Erick Erickson
Arturas: Please make yourself a promise, "Only use the collections commands" ;) At least for a while. Trying to mix collection-level commands and core-level commands is extremely confusing at the start. Under the covers, the Collections API _uses_ the Core API, but in a very precise manner. Any

Re: /replication?command=details does not show infos for all replicas on the core

2018-06-29 Thread Arturas Mazeika
Hi Shawn et al, Thanks a lot for the clarification. It makes a lot of sense and explains which functionality needs to be used to get the infos :-). Out of curiosity: some cores give infos for both shards (through replication query) and some only for one (if you still be able to see the prev

Re: /replication?command=details does not show infos for all replicas on the core

2018-06-29 Thread Shawn Heisey
On 6/29/2018 7:53 AM, Arturas Mazeika wrote: but the query reports infos on only one shard: F:\solr_server\solr-7.2.1>curl -s http://localhost:9996/solr/de_wiki_man/replication?command=details | grep "indexPath\|indexSize" "indexSize":"15.04 GB",

Re: replication

2018-04-13 Thread Shawn Heisey
On 4/10/2018 9:14 AM, Erick Erickson wrote: The very first thing I'd do is set up a simple SolrCloud setup and give it a spin. Unless your indexing load is quite heavy, the added work the NRT replicas have in SolrCloud isn't a problem so worrying about that is premature optimization unless you

Re: replication

2018-04-13 Thread John Blythe
great. thanks, erick! -- John Blythe On Wed, Apr 11, 2018 at 12:16 PM, Erick Erickson wrote: > bq: are you simply flagging the fact that we wouldn't direct the queries > to A > v. B v. C since SolrCloud will make the decisions itself as to which part > of the distro

Re: replication

2018-04-11 Thread Erick Erickson
bq: are you simply flagging the fact that we wouldn't direct the queries to A v. B v. C since SolrCloud will make the decisions itself as to which part of the distro gets hit for the operation Yep. SolrCloud takes care of it all itself. I should also add that there are about a zillion metrics now

Re: replication

2018-04-11 Thread John Blythe
thanks, erick. great info. although you can't (yet) direct queries to one or the other. So just making > them all NRT and forgetting about it is reasonable. are you simply flagging the fact that we wouldn't direct the queries to A v. B v. C since SolrCloud will make the decisions itself as to

Re: replication

2018-04-10 Thread Erick Erickson
bq: should we try to bite the solrcloud bullet and be done w it that's what I'd do. As of 7.0 there are different "flavors", TLOG, PULL and NRT so that's also a possibility, although you can't (yet) direct queries to one or the other. So just making them all NRT and forgetting about it is

Re: replication

2018-04-09 Thread John Blythe
Thanks a bunch for the thorough reply, Shawn. Phew. We’d chosen to go w Master-slave replication instead of SolrCloud per the sudden need we had encountered and the desire to avoid the nuances and changes related to moving to SolrCloud. But so much for this being a more straightforward solution,

Re: replication

2018-04-09 Thread Shawn Heisey
On 4/9/2018 12:15 PM, John Blythe wrote: > we're starting to dive into master/slave replication architecture. we'll > have 1 master w 4 slaves behind it. our app is NRT. if user performs an > action in section A's data they may choose to jump to section B which will > be dependent on having the

Re: Replication in Master Slave Solr setup

2018-03-19 Thread Erick Erickson
The OP was making an invalid assumption I think, that the index would replicate _whenever_ the index changed. But that's not necessarily true, although it's the most common (and default) case. >From the ref guiide: 'If you use "startup", you need to have a "commit" and/or "optimize" entry also

Re: Replication in Master Slave Solr setup

2018-03-18 Thread Shawn Heisey
On 3/17/2018 8:06 PM, vracks wrote: Basic Questions about the Replication in Master Slave Solr Setup. 1) Can Master push the changes to Slaves using the replication handler Replication is always pull -- the slave asks the master if there's anything to copy. 2) If the Answer to the above

Re: Replication Factor Bug in Collections Restore API?

2018-01-05 Thread Ansgar Wiechers
On 2018-01-04 Shalin Shekhar Mangar wrote: > Sounds like a bug. Can you please open a Jira issue? https://issues.apache.org/jira/browse/SOLR-11823 Regards Ansgar Wiechers

Re: Replication Factor Bug in Collections Restore API?

2018-01-04 Thread Shalin Shekhar Mangar
Sounds like a bug. Can you please open a Jira issue? On Thu, Jan 4, 2018 at 8:37 PM, Ansgar Wiechers wrote: > Hi all. > > I'm running Solr 7.1 in SolrCloud mode ona a 3-node cluster and tried > using the backup/restore API for the first time. Backup worked fine, but >

Re: Replication on startup takes a long time

2017-09-25 Thread Erick Erickson
Emir: OK, thanks for pointing that out, that relieves me a lot! Erick On Mon, Sep 25, 2017 at 1:03 AM, Emir Arnautović wrote: > Hi Eric, > I don’t think that there are some bugs with searcher reopening - this is a > scenario with a new slave: > > “But when I add

Re: Replication on startup takes a long time

2017-09-25 Thread Emir Arnautović
Hi Eric, I don’t think that there are some bugs with searcher reopening - this is a scenario with a new slave: “But when I add a *new* slave pointing to the master…” So expected to have zero results until replication finishes. Regards, Emir > On 23 Sep 2017, at 19:21, Erick Erickson

Re: Replication on startup takes a long time

2017-09-23 Thread Erick Erickson
First I'd like to say that I wish more people would take the time like you have to fully describe the problem and your observations, it makes it s much nicer than having half-a-dozen back and forths! Thanks! Just so it doesn't get buried in the rest of the response, I do tend to go on I

Re: Replication Question

2017-08-04 Thread Shawn Heisey
On 8/2/2017 8:56 AM, Michael B. Klein wrote: > SCALE DOWN > 1) Call admin/collections?action=BACKUP for each collection to a > shared NFS volume > 2) Shut down all the nodes > > SCALE UP > 1) Spin up 2 Zookeeper nodes and wait for them to stabilize > 2) Spin up 3 Solr nodes and wait for them to

Re: Replication Question

2017-08-02 Thread Michael B. Klein
And the one that isn't getting the updates is the one marked in the cloud diagram as the leader. /me bangs head on desk On Wed, Aug 2, 2017 at 10:31 AM, Michael B. Klein wrote: > Another observation: After bringing the cluster back up just now, the > "1-in-3 nodes don't get

Re: Replication Question

2017-08-02 Thread Michael B. Klein
Another observation: After bringing the cluster back up just now, the "1-in-3 nodes don't get the updates" issue persists, even with the cloud diagram showing 3 nodes, all green. On Wed, Aug 2, 2017 at 9:56 AM, Michael B. Klein wrote: > Thanks for your responses, Shawn and

Re: Replication Question

2017-08-02 Thread Michael B. Klein
Thanks for your responses, Shawn and Erick. Some clarification questions, but first a description of my (non-standard) use case: My Zookeeper/SolrCloud cluster is running on Amazon AWS. Things are working well so far on the production cluster (knock wood); its the staging cluster that's giving

Re: Replication Question

2017-08-01 Thread Erick Erickson
And please do not use optimize unless your index is totally static. I only recommend it when the pattern is to update the index periodically, like every day or something and not update any docs in between times. Implied in Shawn's e-mail was that you should undo anything you've done in terms of

Re: Replication Question

2017-08-01 Thread Shawn Heisey
On 8/1/2017 12:09 PM, Michael B. Klein wrote: > I have a 3-node solrcloud cluster orchestrated by zookeeper. Most stuff > seems to be working OK, except that one of the nodes never seems to get its > replica updated. > > Queries take place through a non-caching, round-robin load balancer. The >

Re: Replication Index fetch failed

2016-10-10 Thread Arkadi Colson
Hi I could not find "Could not download file" in the logs. Should I increase the log level somewhere? Just let me know... so I can provide you more detailed logs... Thx! Arkadi On 02-09-16 11:21, Arkadi Colson wrote: Hi I cannot find a string in the logs matching "Could not download

Re: Replication Index fetch failed

2016-09-02 Thread Arkadi Colson
Hi I cannot find a string in the logs matching "Could not download file...". This info is logged on the slave: WARN - 2016-09-02 09:28:36.923; [c:intradesk s:shard10 r:core_node23 x:intradesk_shard10_replica1] org.apache.solr.handler.IndexFetcher$FileFetcher; Error in fetching file:

Re: Replication Index fetch failed

2016-09-01 Thread Shalin Shekhar Mangar
On Thu, Sep 1, 2016 at 6:05 PM, Arkadi Colson wrote: > ERROR - 2016-09-01 14:30:43.653; [c:intradesk s:shard1 r:core_node5 > x:intradesk_shard1_replica1] org.apache.solr.common.SolrException; Index > fetch failed :org.apache.solr.common.SolrException: Unable to download >

Re: Replication with managed resources?

2016-08-04 Thread rosbaldeston
Raised as https://issues.apache.org/jira/browse/SOLR-9382 -- View this message in context: http://lucene.472066.n3.nabble.com/Replication-with-managed-resources-tp4289880p4290386.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Replication with managed resources?

2016-08-03 Thread rosbaldeston
I was just running my own test and it seems it doesn't replicate or reload the managed schema synonyms file. Not on a manual replication request after a synonym change and not on an index change triggering an automatic replication at least. Used this as the slaves confFiles, not sure if this

Re: Replication with managed resources?

2016-08-03 Thread Erick Erickson
bq: I'm also guessing those _schema and managed_schema files are an implementation detail for the missing zookeeper functionality. But if I did add those to a conffiles option it might automate the slave core reloads for me? You're getting closer ;). There's nothing Cloud specific about the whole

Re: Replication with managed resources?

2016-08-03 Thread rosbaldeston
Erick Erickson wrote > It Depends. When running in Cloud mode then "yes". If you're running > stand-alone > then there is no Zookeeper running so the answer is "no". Ah that helps, so no zookeeper in my case. I did wonder if it wasn't just sharing the same config files between master and slave

Re: Replication with managed resources?

2016-08-03 Thread Erick Erickson
bq: Am I right in saying managed resources are handled by zookeeper rather than files on the filesystem It Depends. When running in Cloud mode then "yes". If you're running stand-alone then there is no Zookeeper running so the answer is "no". You can run Solr just like you always have in

Re: Replication with managed resources?

2016-08-03 Thread rosbaldeston
Am I right in saying managed resources are handled by zookeeper rather than files on the filesystem and I should ignore any files such as: managed-schema, _rest_managed.json, _schema_analysis_stopwords_english.json, _schema_analysis_synonyms_english.json ... I should not try to copy any of

Re: Replication as backup in SolrCloud

2015-11-15 Thread KNitin
We built and open sourced haft precisely for such use cases. https://github.com/bloomreach/solrcloud-haft You can clone an entire cluster or selective collections between clusters. It has only been tested upto solr 4.10. Let me know if you run into

Re: Replication as backup in SolrCloud

2015-11-15 Thread Arcadius Ahouansou
Hello Gr. We are in a similar situation to yours... and we are using https://issues.apache.org/jira/browse/SOLR-8146 It is a small patch for the SolrJ client that can send all of your queries to your main DC unless all nodes in the main DC are down. Write/updates/delete and admin operations

Re: Replication and soft commits for NRT searches

2015-10-15 Thread MOIS Martin (MORPHO)
Hello, the background for my question is that one of the requirements for our injection tool is that it should report that a new document has been successfully enrolled to the cluster only if it is available on all replicas. The automated integration test for this feature will submit a

Re: Replication and soft commits for NRT searches

2015-10-15 Thread Erick Erickson
bq: the background for my question is that one of the requirements for our injection tool is that it should report that a new document has been successfully enrolled to the cluster only if it is available on all replicas Frankly, this is the tail wagging the dog. SolrCloud is designed to

Re: Replication and soft commits for NRT searches

2015-10-14 Thread Erick Erickson
bq: If a timeout between shard leader and replica can lead to a smaller rf value (because replication has timed out), is it possible to increase this timeout in the configuration? Why do you care? If it timed out, then the follower will no longer be active and will not serve queries. The Cloud

Re: Replication and soft commits for NRT searches

2015-10-14 Thread MOIS Martin (MORPHO)
Hello, thank you for the detailed answer. If a timeout between shard leader and replica can lead to a smaller rf value (because replication has timed out), is it possible to increase this timeout in the configuration? Best Regards, Martin Mois Comments inline: On Mon, Oct 12, 2015 at 1:31

Re: Replication and soft commits for NRT searches

2015-10-13 Thread Shalin Shekhar Mangar
Comments inline: On Mon, Oct 12, 2015 at 1:31 PM, MOIS Martin (MORPHO) wrote: > Hello, > > I am running Solr 5.2.1 in a cluster with 6 nodes. My collections have been > created with replicationFactor=2, i.e. I have one replica for each shard. > Beyond that I am using

Re: Replication and soft commits for NRT searches

2015-10-12 Thread Erick Erickson
First of all, setting soft commit with maxDocs=1 is almost (but not quite) guaranteed to lead to problems. For _every_ document you add to Solr, all your top-level caches (i.e. the ones configured in solrconrig.xml) will be thrown away, all autowarming will be performed etc. Essentially assuming a

Re: Replication Sync OR Async?

2015-09-09 Thread Shawn Heisey
On 9/8/2015 11:16 PM, Maulin Rathod wrote: > When replicas are running it took around 900 seconds for indexing. > After stopping replicas it took around 500 seconds for indexing. > > Is the replication happens in Sync or Async? If it is Sync, can we make it > Async so that it will not affect

RE: Replication Sync OR Async?

2015-09-09 Thread Maulin Rathod
Hi Shawn, Thanks for reply. If we keep replication Async, Can error handling not work same like replica down scenario? -Original Message- From: Shawn Heisey [mailto:apa...@elyograg.org] Sent: 09 September 2015 19:40 To: solr-user@lucene.apache.org Subject: Re: Replication Sync OR Async

Re: replication and HDFS

2015-08-31 Thread Joseph Obernberger
Thank you Erick. What about cache size? If we add replicas to our cluster and each replica has nGBytes of RAM allocated for HDFS caching, would that help performance? Specifically the performance we want to increase is time to facet data, time to cluster data and search time. While we

Re: replication and HDFS

2015-08-31 Thread Erick Erickson
Yes, No, Maybe. bq; Specifically the performance we want to increase is time to facet data, time to cluster data and search time Well, that about covers everything ;) You cannot talk about this without also taking about cache warming. Given your setup, I'm guessing you have very few searches on

Re: replication and HDFS

2015-08-20 Thread Erick Erickson
Yes. Maybe. It Depends (tm). Details matter (tm). If you're firing just a few QPS at the system, then improved throughput by adding replicas is unlikely. OTOH, if you're firing lots of simultaneous queries at Solr and are pegging the processors, then adding replication will increase aggregate

Re: Replication as backup in SolrCloud

2015-06-22 Thread Erick Erickson
Currently, one is best off treating these as two separate clusters and having your client send the data to both, or reproducing your system-of-record and running your DCs completely separately. Hopefully soon, though, there'll be what you're asking for active/passive DCs, see:

Re: Replication for SolrCloud

2015-04-19 Thread gengmao
Thanks for the suggestion, Erick. However here what we need is not a patch, is a clarification from practice perspective. I think solr replication is a great feature to scale reads, and kind of increase reliability. However, on HDFS it is not as useful as just sharding. Sharding can scale both

Re: Replication for SolrCloud

2015-04-19 Thread juergen.wag...@devoteam.com
In simple words: HDFS is good for file-oriented replication. Solr is good for index replication. Consequently, if atomic file update operations of an application (like Solr) are not atomic on a file level, HDFS is not adequate - like for Solr with live index updates. Running Solr on HDFS (as a

Re: Replication for SolrCloud

2015-04-19 Thread gengmao
Please see my response in line: On Fri, Apr 17, 2015 at 10:59 PM Shalin Shekhar Mangar shalinman...@gmail.com wrote: Some comments inline: On Sat, Apr 18, 2015 at 2:12 PM, gengmao geng...@gmail.com wrote: On Sat, Apr 18, 2015 at 12:20 AM Jürgen Wagner (DVT) juergen.wag...@devoteam.com

Re: Replication for SolrCloud

2015-04-18 Thread Erick Erickson
AFAIK, the HDFS replication of Solr indexes isn't something that was designed, it just came along for the ride given HDFS replication. Having a shard with 1 leader and two followers have 9 copies of the index around _is_ overkill, nobody argues that at all. I know the folks at Cloudera (who

Re: Replication for SolrCloud

2015-04-18 Thread gengmao
On Sat, Apr 18, 2015 at 12:20 AM Jürgen Wagner (DVT) juergen.wag...@devoteam.com wrote: Replication on the storage layer will provide a reliable storage for the index and other data of Solr. In particular, this replication does not guarantee your index files are consistent at any time as

Re: Replication for SolrCloud

2015-04-18 Thread gengmao
I wonder why need to use SolrCloud replication on HDFS at all, given HDFS already provides replication and availability? The way to optimize performance and scalability should be tweaking shards, just like tweaking regions on HBase - which doesn't provide region replication too, isn't it? I have

Re: Replication for SolrCloud

2015-04-18 Thread Shalin Shekhar Mangar
Some comments inline: On Sat, Apr 18, 2015 at 2:12 PM, gengmao geng...@gmail.com wrote: On Sat, Apr 18, 2015 at 12:20 AM Jürgen Wagner (DVT) juergen.wag...@devoteam.com wrote: Replication on the storage layer will provide a reliable storage for the index and other data of Solr. In

Re: Replication for SolrCloud

2015-04-09 Thread Erick Erickson
Yes. 3 replicas and an HDFS replication factor of 3 means 9 copies of the index are laying around. You can change your HDFS replication factor, but that affects other applications using HDFS, so that may not be an option. Best, Erick On Thu, Apr 9, 2015 at 2:31 AM, Vijaya Narayana Reddy Bhoomi

Re: Replication of a corrupt master index

2014-12-02 Thread Erick Erickson
No. The master is the master and will always stay the master unless you change it. This is one of the reasons I really like to keep the original source around in case I every have this problem. Best, Erick On Tue, Dec 2, 2014 at 2:34 AM, Charra, Johannes

Re: Replication of a corrupt master index

2014-12-02 Thread Erick Erickson
-Ursprüngliche Nachricht- Von: Erick Erickson [mailto:erickerick...@gmail.com] Gesendet: Dienstag, 2. Dezember 2014 15:54 An: solr-user@lucene.apache.org Betreff: Re: Replication of a corrupt master index No. The master is the master and will always stay the master unless you change it. This is one

Re: Replication of full index to replica after merge index into leader not working

2014-08-19 Thread Mark Miller
I’d just file a JIRA. Merge, like optimize and a few other things, were never tested or considered in early SolrCloud days. It’s used in the HDFS stuff, but in that case, the index is merged to all replicas and no recovery is necessary. If you want to make the local filesystem merge work well

Re: Replication of full index to replica after merge index into leader not working

2014-08-19 Thread Mark Miller
On August 19, 2014 at 1:33:10 PM, Mark Miller (markrmil...@gmail.com) wrote: sounds like we should write a test and make it work. Keeping in mind that when using a shared filesystem like HDFS or especially if using the MapReduce contrib, you probably won’t want this new behavior. --  Mark

Re: Replication of full index to replica after merge index into leader not working

2014-08-19 Thread Timothy Potter
Was able to get around it for now sending the REQUESTRECOVERY command to the replica. Will open an improvement JIRA but not sure if it's worth it as the work-around is pretty clean (IMO). Tim On Tue, Aug 19, 2014 at 5:33 PM, Mark Miller markrmil...@gmail.com wrote: I’d just file a JIRA. Merge,

RE: Replication Issue with Repeater Please help

2014-08-16 Thread waqas sarwar
Date: Thu, 14 Aug 2014 06:51:02 -0600 From: s...@elyograg.org To: solr-user@lucene.apache.org Subject: Re: Replication Issue with Repeater Please help On 8/14/2014 2:09 AM, waqas sarwar wrote: Thanks Shawn. What i got is Circular replication is totally impossible Solr fails

Re: Replication Issue with Repeater Please help

2014-08-16 Thread Erick Erickson
: Re: Replication Issue with Repeater Please help On 8/14/2014 2:09 AM, waqas sarwar wrote: Thanks Shawn. What i got is Circular replication is totally impossible Solr fails in distributed environment. Then why solr documentation says that configure REPEATER for distributed architecture

Re: Replication Issue with Repeater Please help

2014-08-16 Thread Shawn Heisey
On 8/16/2014 8:11 AM, waqas sarwar wrote: Thank you so much. You helped alot. One more question is that can i use only one zookeeper server to manage 3 solr servers, or i've to configure 3 zookeeper servers for each. And zookeeper servers should be stand alone or better to use same solr

RE: Replication Issue with Repeater Please help

2014-08-14 Thread waqas sarwar
Date: Wed, 13 Aug 2014 07:19:58 -0600 From: s...@elyograg.org To: solr-user@lucene.apache.org Subject: Re: Replication Issue with Repeater Please help On 8/13/2014 12:49 AM, waqas sarwar wrote: Hi, I'm using Solr. I need a little bit assistance from you. I am bit stuck

Re: Replication Issue with Repeater Please help

2014-08-14 Thread Shawn Heisey
On 8/14/2014 2:09 AM, waqas sarwar wrote: Thanks Shawn. What i got is Circular replication is totally impossible Solr fails in distributed environment. Then why solr documentation says that configure REPEATER for distributed architecture, because REPEATER behave like master-slave at a

Re: Replication Issue with Repeater Please help

2014-08-13 Thread Shawn Heisey
On 8/13/2014 12:49 AM, waqas sarwar wrote: Hi, I'm using Solr. I need a little bit assistance from you. I am bit stuck with Solr replication, before discussing issue let me write a brief description.Scenario:- I want to set up solr in distributed architecture, suppose start with

Re: Replication Problem from solr-3.6 to solr-4.0

2014-07-24 Thread Sree..
I did optimize the master and the slave started replicating the indices! -- View this message in context: http://lucene.472066.n3.nabble.com/Replication-Problem-from-solr-3-6-to-solr-4-0-tp4025028p4148953.html Sent from the Solr - User mailing list archive at Nabble.com.

Re: Replication Problem from solr-3.6 to solr-4.0

2014-07-22 Thread askumar1444
Same with me too, in a multi-core Master/Slave. 11:17:30.476 [snapPuller-8-thread-1] INFO o.a.s.h.SnapPuller - Master's generation: 87 11:17:30.476 [snapPuller-8-thread-1] INFO o.a.s.h.SnapPuller - Slave's generation: 3 11:17:30.476 [snapPuller-8-thread-1] INFO o.a.s.h.SnapPuller - Starting

Re: Replication (Solr Cloud)

2014-03-25 Thread Shawn Heisey
On 3/25/2014 10:42 AM, Software Dev wrote: I see that by default in SolrCloud that my collections are replicating. Should this be disabled in SolrCloud as this is already handled by it? From the documentation: The Replication screen shows you the current replication state for the named core

Re: Replication (Solr Cloud)

2014-03-25 Thread Michael Della Bitta
No, don't disable replication! The way shards ordinarily keep up with updates is by sending every document to each member of the shard. However, if a shard goes offline for a period of time and comes back, replication is used to catch up that shard. So you really need it on. If you created your

Re: Replication (Solr Cloud)

2014-03-25 Thread Software Dev
Thanks for the reply. Ill make sure NOT to disable it.

  1   2   3   4   5   >