Re: Ingestion not scaling horizontally as I add more cores to Solr

ith a group once that used laptops to benchmark a service and it was a WiFi network limit that caused weird results. LAN connections or even better a dedicated client machine would help push more documents. Kevin Risden On Thu, Jan 11, 2018 at 11:39 AM, Shashank Pedama

Re: Ingestion not scaling horizontally as I add more cores to Solr

is with the client side. Thanks, Shashank On 1/10/18, 10:54 PM, "Shawn Heisey" <apa...@elyograg.org> wrote: On 1/10/2018 12:58 PM, Shashank Pedamallu wrote: > As you can see, the number of documents being ingested per core is not scaling horizontally as I'm adding more core

Re: Ingestion not scaling horizontally as I add more cores to Solr

t. Best, Erick On Wed, Jan 10, 2018 at 1:52 PM, Shashank Pedamallu <spedama...@vmware.com> wrote: > - Did you sept up an actual multiple node cluster or are you running this > all on one box? > Sorry, I should have mentioned this earlier

Re: Ingestion not scaling horizontally as I add more cores to Solr

solr cluster... On Wed, Jan 10, 2018 at 2:58 PM, Shashank Pedamallu <spedama...@vmware.com> wrote: > Hi, > > > > I’m trying to find the upper thresholds of ingestion and I have tried the > following. In each of the experiments, I’m

Ingestion not scaling horizontally as I add more cores to Solr

Hi, I’m trying to find the upper thresholds of ingestion and I have tried the following. In each of the experiments, I’m ingesting random documents with 5 fields. Number of Cores Number of documents ingested per second per core 1 89000 3 33000 5 18000 As you can see, the

Re: Core size - distinguish between merge and deletes

ources reclaimed, i.e. the index gets smaller. You can figure out what % of your index consists of deleted documents by the delta between numDocs and maxDocs, available on the admin UI and from the Luke handler (and maybe JMX). Best, Erick On Thu, Nov 9, 2017 at 12

Core size - distinguish between merge and deletes

Hi, I wanted to get accurate metrics regarding to the amount of data being indexed in Solr. In this regard, I observe that sometimes, this number decreases due to lucene merges. But I’m also deleting data at times. Is there a way to distinguish between when size is being reduced because of a

Time to Load a Solr Core with Hdfs Directory Factory

Hi, I’m trying an experiment in which, I’m loading a core of 1.27GB with 5621600 documents on 2 Solr setups. On first setup, dataDir is pointed as a NRTCachingDirectory as a standard path in my local. On second setup, it is pointed as HdfsDirectory. As part of loading the core, I see following

Re: Error Opening new IndexSearcher - LockObtainFailedException

iyiQLSZS38Qe-ZLj-2OeN-u0= On Sep 21, 2017 10:19 PM, "Shashank Pedamallu" <spedama...@vmware.com> wrote: Hi, I’m seeing the following exception in Solr that gets automatically resolved eventually. 2017-09-22 00:18:17.243 ERROR (qtp1702660

Error Opening new IndexSearcher - LockObtainFailedException

Hi, I’m seeing the following exception in Solr that gets automatically resolved eventually. 2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1] o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error opening new searcher

How Solr knows the Cores it has on startup?

Hi, I wanted to know how does Solr pick up cores on startup. Basically, what I would like to try is Read cores created from one Solr instance from another Solr instance. i.e., * I will have 2 Solr Instances SOLR1, SOLR2. Only SOLR1 is started. * I’m creating a core (Core1) using SOLR1.

Solr with Alluxio

Hi, I’m trying to access index files of a core stored in S3 by using Alluxio with Solr. Did anyone try integrating Solr with Alluxio? Can someone give me any pointers? Also, I’ve come across this session in the Solr Conference around next week -

Re: Time taken for CheckIndex on a core

Thanks a lot for the reply Shawn! That makes sense. Thanks, Shahsank On 8/17/17, 11:37 AM, "Shawn Heisey" <apa...@elyograg.org> wrote: On 8/14/2017 8:19 PM, Shashank Pedamallu wrote: > I was looking for a reliable method to check if my core was in a > con

Time taken for CheckIndex on a core

Hi, I was looking for a reliable method to check if my core was in a consistent readable state and came across org.apache.lucene.index.CheckIndex class. However, when I tried to run this command multiple times from the command-line via the main utility method provided, it takes long time to

Get commit time of Solr on closing of a core

Hi, I observed that in Solr, when a core closes, it does a commit on the core, but does not make post commit callBacks. In this scenario, is there a reliable way to determine the lastCommitTime of a core in the closeHook of a core handler? Thanks, Shashank

Data Dir from a core with init-failure

data Step 6: Repeat Step 1. I.e., remove some index file and restart Solr Step 7: Repeat Step 2. I.e, Invoke status api. Response was: { "responseHeader":{ "status":0, "QTime":0}, "initFailures":{"":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Error opening new searcher"}, "status":{ "":{}}} Like you see, the response of status api from Step 7 is different from Step 2. Is there a reliable way to get dataDir of a core in the presence of init-failure? Thanks, Shashank Pedamallu

Re: Verify unloading of a Solr Core

then? Please note that, I’m making these requests from a CoreAdminHandler. Thanks, Shashank Pedamallu On 5/3/17, 1:09 PM, "Shashank Pedamallu" <spedama...@vmware.com> wrote: >Please ignore this question. I’m not able to reproduce this, it should have >been a mi

Re: Verify unloading of a Solr Core

Please ignore this question. I’m not able to reproduce this, it should have been a mistake by me. Thanks, Shashank Pedamallu On 5/3/17, 12:52 PM, "Shashank Pedamallu" <spedama...@vmware.com> wrote: >Hi all, > >I unload a core from Solr using the CoreAdmin

Verify unloading of a Solr Core

}} At this step, I had put a debug point and Solr UI still shows the core that I have unloaded. Also, I killed the solr process using kill –9 command and restarted Solr. The core is still present in Solr. When does it get unloaded then? Thanks in advance! Thanks, Shashank Pedamallu

Re: Reload an unloaded core

Pedamallu On 5/2/17, 2:51 PM, "Shawn Heisey" <apa...@elyograg.org> wrote: >On 5/2/2017 10:53 AM, Shashank Pedamallu wrote: >> I want to unload a core from Solr without deleting data-dir or instance-dir. >> I’m performing some operations on the data-dir after

Reload an unloaded core

no such core exists. 2. Create api – throws an exception saying a core with given name already exists. Can someone point me what api I could use to achieve this. Please note that, I’m working with Solr in Non-Cloud mode without Zookeeper, Collections, etc. Thanks in advance! Thanks, Shashank Pedamallu

Re: Maintaining variable values between transition states

particularly the >CoreDescriptor (which is the in-memory form of core.properties) has >some issues with multiple copies being kept around so directly writing >core.properties has some gotchas currently that should be fixed in the >above. > >Best, >Erick > > >On Thu, Mar 3

Maintaining variable values between transition states

Hi All, I have some configuration variables that I need to hold in Solr as it switches between transient states on a transient core. What is the best way to do this? These variables can change value during a running environment. So, I need to have read and write access to the persistent store.

Re: Avoiding Transient state during a long running background process

Hi, Thanks Erik and Shawn for so much info! 1) Yes, I have deliberately put transientCacheSize as very small(2) in my dev laptop for testing how Solr handles the switch and how it affects my backup process. 2) Yes, I'm not using Solr Cloud. Each individual Solr core is independent and we are

Re: Avoiding Transient state during a long running background process

such processes. Does ReplicationHandler not get affected if a replication job is in progress and Solr decided to unload this core in between the process. Thanks, Shashank On 3/29/17, 4:41 PM, "Shawn Heisey" <apa...@elyograg.org> wrote: >On 3/29/2017 4:50 PM, Shash

Re: Avoiding Transient state during a long running background process

Hi Shawn, Thank you very much for the response. Is there no definite way of ensuring that Solr does not switch transient states by an api? Like solrCore.open() and solrCore.close()? Thanks, Shashank Pedamallu MTS MBU vCOps Dev US-CA-Promontory E, E 1035 Email: spedama...@vmware.com Office

Avoiding Transient state during a long running background process

Hi, I’m performing some long running operation on a Background thread on a Core and I observed that since the core has the property “transient” set to true, in between this operation completes, the core is being CLOSED and OPENED by Solr (even though the operation continues without

Managing between or avoiding Transient state for specific period of time in Solr

interruption). Is there a way to avoid/block OPENING/CLOSING of the core when this operation is in progress? Thanks, Shashank Pedamallu MTS MBU vCOps Dev US-CA-Promontory E, E 1035 Email: spedama...@vmware.com Office: 650.427.6280 x76280

Post Body from RequestHandler in Solr

Hi all, This might be a very basic question, but I’m struggling to find an answer. How can I access the json I passed as a Post body within Solr RequestHandler. I’m getting the Url Params using SolrQueryRequest.getParams(), but I’m not able to figure out how to get Post Body. Could not find an

Loading Third party libraries along with Solr

(amazon-aws third party libs) appropriately to not cause issue with the rest of the jars. Thanks, Shashank Pedamallu