Did you try force merging, given that your indices are static? On Thu, Aug 8, 2019 at 12:22 PM Aravind S (User Intent) <s.arav...@flipkart.com> wrote: > > Hi Atri, > > Thanks for the prompt reply. > > There are no deletion or updation we are trying to merge offline generated > sorted indices into a single sorted segment index. This entire process is > completely offline and there is no online serving for this index taking no > live updates like addition or deletion. > > Each index is of size is close to 1 GB. We are trying to merge 25 such > indices into a single index using add indexes of Lucene IndexWriter using > ConcurrentMergeScheduler. > > Is there any recommendation on how to go about merging multiple sorted > indices into a single index more efficiently? > > Regards, > Aravind S > > On Thu, Aug 8, 2019, 11:57 AM Atri Sharma <a...@apache.org> wrote: >> >> Have you tried a more frequent merging? What are the average sizes of >> your segments, and what does your deletion/updates rate look like? >> >> On Thu, Aug 8, 2019 at 1:39 AM Aravind S (User Intent) >> <s.arav...@flipkart.com> wrote: >> > >> > Hi, >> > >> > We are currently trying to merge sorted indices in an offline process. >> > This process is taking a lot of time in merging. We tried using >> > ConcurrentMergeScheduler with tiered merge policy. >> > >> > We see the merger threads to be maximum to be set as 4 in >> > ConcurrentMergerScheduler setDefaultMaxMergesAndThreads method. >> > >> > Is there a way to decrease the time in merging this sorted indices. Is >> > there any recommendation that could be followed to scale merging with an >> > increase in a number of indices to be merged. >> > >> > Regards, >> > Aravind S >> > >> > ----------------------------------------------------------------------------------------- >> > >> > This email and any files transmitted with it are confidential and intended >> > solely for the use of the individual or entity to whom they are addressed. >> > If you have received this email in error, please notify the system >> > manager. This message contains confidential information and is intended >> > only for the individual named. If you are not the named addressee, you >> > should not disseminate, distribute or copy this email. Please notify the >> > sender immediately by email if you have received this email by mistake and >> > delete this email from your system. If you are not the intended recipient, >> > you are notified that disclosing, copying, distributing or taking any >> > action in reliance on the contents of this information is strictly >> > prohibited. >> > >> > >> > >> > Any views or opinions presented in this email are solely those of the >> > author and do not necessarily represent those of the organization. Any >> > information on shares, debentures or similar instruments, recommended >> > product pricing, valuations and the like are for information purposes >> > only. It is not meant to be an instruction or recommendation, as the case >> > may be, to buy or to sell securities, products, services nor an offer to >> > buy or sell securities, products or services unless specifically stated to >> > be so on behalf of the Flipkart group. Employees of the Flipkart group of >> > companies are expressly required not to make defamatory statements and not >> > to infringe or authorise any infringement of copyright or any other legal >> > right by email communications. Any such communication is contrary to >> > organizational policy and outside the scope of the employment of the >> > individual concerned. The organization will not accept any liability in >> > respect of such communication, and the employee responsible will be >> > personally liable for any damages or other liability arising. >> > >> > >> > >> > Our organization accepts no liability for the content of this email, or >> > for the consequences of any actions taken on the basis of the information >> > provided, unless that information is subsequently confirmed in writing. If >> > you are not the intended recipient, you are notified that disclosing, >> > copying, distributing or taking any action in reliance on the contents of >> > this information is strictly prohibited. >> > >> > ----------------------------------------------------------------------------------------- >> >> -- >> Regards, >> >> Atri >> Apache Concerted >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> > > ----------------------------------------------------------------------------------------- > > This email and any files transmitted with it are confidential and intended > solely for the use of the individual or entity to whom they are addressed. If > you have received this email in error, please notify the system manager. This > message contains confidential information and is intended only for the > individual named. If you are not the named addressee, you should not > disseminate, distribute or copy this email. Please notify the sender > immediately by email if you have received this email by mistake and delete > this email from your system. If you are not the intended recipient, you are > notified that disclosing, copying, distributing or taking any action in > reliance on the contents of this information is strictly prohibited. > > > > Any views or opinions presented in this email are solely those of the author > and do not necessarily represent those of the organization. Any information > on shares, debentures or similar instruments, recommended product pricing, > valuations and the like are for information purposes only. It is not meant to > be an instruction or recommendation, as the case may be, to buy or to sell > securities, products, services nor an offer to buy or sell securities, > products or services unless specifically stated to be so on behalf of the > Flipkart group. Employees of the Flipkart group of companies are expressly > required not to make defamatory statements and not to infringe or authorise > any infringement of copyright or any other legal right by email > communications. Any such communication is contrary to organizational policy > and outside the scope of the employment of the individual concerned. The > organization will not accept any liability in respect of such communication, > and the employee responsible will be personally liable for any damages or > other liability arising. > > > > Our organization accepts no liability for the content of this email, or for > the consequences of any actions taken on the basis of the information > provided, unless that information is subsequently confirmed in writing. If > you are not the intended recipient, you are notified that disclosing, > copying, distributing or taking any action in reliance on the contents of > this information is strictly prohibited. > > -----------------------------------------------------------------------------------------
-- Regards, Atri Apache Concerted --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
