Nice +1 Le ven. 7 juil. 2023 à 16:07, David Smiley <david.w.smi...@gmail.com> a écrit :
> I gave it a look. I like it! > > ~ David > > > On Thu, Jul 6, 2023 at 6:22 PM Pierre Salagnac <pierre.salag...@gmail.com> > wrote: > > > Here is my POC to add a queue into CoreAdminHandler: > > https://github.com/apache/solr/pull/1761 > > > > It does the following: > > - add a flag to core admin operations to be marked as expensive. For now, > > only backup and restore are expensive, this may be extended. > > - in CoreAdminHandler, we count the number of in-flight expensive > > operations. If more than the limit (currently 5 by default) are > > already in-flight, we don't submit any new ones to the thread pool, but > we > > add them into a queue. > > - each time an expensive operation is completed, it starts the following > > one from the queue, if any. > > > > Let me know what you think > > Thanks > > > > Le jeu. 29 juin 2023 à 15:37, Pierre Salagnac <pierre.salag...@gmail.com > > > > a > > écrit : > > > > > Jason, I haven't done much scalability testing, so it's hard to give > > > accurate numbers on when we start having issues. > > > For the environment I looked in detail we run a 16 nodes cluster, and > the > > > collection I wasn't able to backup has about 1500 shards, ~1.5 GB each. > > > > > > Core backups/restores are expensive calls, compared to other admin > calls > > > like an empty core creation. I'm not sure of the full list of expensive > > and > > > non-expense operations, but we may also have a fairness issue when > > > expensive operations block the non expensive ones for a while. > > > > > > I had a great discussion with David. > > > I see two options for a long term fix: > > > > > > 1. Throttle core backups/restore at the top level (in the overseer). > > > My approach was to not send too many concurrent requests from > BackupCmd. > > > I have a decent POC for backups, but it should be refactored to share > > this > > > mechanism for all admin operations. > > > It will be harder to achieve in distributed mode (no overseer), since > > > we'll need a central place to count how many backups are in-flight. We > > may > > > somehow lock in Zookeeper for this, so it may be over complex at the > end. > > > > > > > > > 2. Throttle in each node. > > > - Currently, all async admin operations are handled by a > > > ThreadPoolExecutor defined in CoreAdminHandler class. This pool is > > > hardcoded with a size of 50, so if we receive more than 50 concurrent > > > tasks, we add them in the queue of the executor. For a large collection > > > backup, each node immediately starts 50 concurrent core snapshots. > > > - Instead of immediately submitting to the executor, I think we should > > > manage our own queue here for expensive operations. By counting the > > number > > > of in-flight backups/restores, we don't submit to the executor more > than > > > (lets say) 10 concurrent backups. Each time one is complete, we poll > the > > > queue and start the next one if appropriate. > > > - This ensures fairness for expensive and non expensive operations. > > > Non-expensive ones will always have at least 40 threads to be quickly > > > handled. And this works well in distributed mode since the overseer is > > not > > > involved. > > > - This could be extended to define more than one queue, but I'm not > sure > > > it is worth it. > > > > > > Pierre > > > > > > > > > Le mar. 27 juin 2023 à 19:16, David Smiley <dsmi...@apache.org> a > écrit > > : > > > > > >> Here's a POC: https://github.com/apache/solr/pull/1729 > > >> > > >> ~ David Smiley > > >> Apache Lucene/Solr Search Developer > > >> http://www.linkedin.com/in/davidwsmiley > > >> > > >> > > >> On Mon, Jun 19, 2023 at 3:36 PM David Smiley <dsmi...@apache.org> > > wrote: > > >> > > >> > Has anyone mitigated the potentially large IO impact of doing a > backup > > >> of > > >> > a large collection or just in general? If the collection is large > > >> enough, > > >> > there very well could be many shards on one host and it could > saturate > > >> the > > >> > IO. I wonder if there should be a rate limit mechanism or some > other > > >> > mechanism. > > >> > > > >> > Not the same but I know that at a segment level, the merges are rate > > >> > limited -- ConcurrentMergeScheduler doesn't quite let you set it but > > >> > adjusts itself automatically ("ioThrottle" boolean). > > >> > > > >> > ~ David Smiley > > >> > Apache Lucene/Solr Search Developer > > >> > http://www.linkedin.com/in/davidwsmiley > > >> > > > >> > > > > > >
