On Thu, Feb 1, 2024 at 1:53 PM Ilan Ginzburg <ilans...@gmail.com> wrote: > > I'd be in favor of the Overseer dropping synchronous requests for which the > requestor is no longer waiting (ephemeral ZK node is gone).
I agree! As you know, we've customized Solr to do exactly that for collection creation. We suspect a misaligned timeout kept the requestor/client present from Solr's perspective even though they actually gave up in some other thread on their end and this wasn't cancelled/stopped. Still; the proposal here seems more resilient than fixing that; it's debatable. And if implemented; it may obsolete the ephemeral node check in practice even though they are complementary. > For sync or async requests, we could let the caller set a timeout after > which the processing should not start if it hasn't already, I thought it'd be nice to avoid a new per-request message and instead use a node-wide setting. A per-request setting would show up in basically every API definition in Solr's new OpenAPI that we machine generate -- solr/api/build/generated/openapi/*.json -- as does "async" already. I don't think it's worth all the API littering considering the Overseer mode (vs distributed processing) is what this concern applies to, it's maybe a niche issue, and I hope to see the new distributed mode used more and more. > or for async > messages allow a cancellation call (that would cancel if processing has not > started). This part works that way already I think. > Once processing has started, I suggest we let it finish (cancelling > processing in progress would be more complicated). Yeah I'm not proposing anything to the contrary. > Ilan > > On Thu, Feb 1, 2024 at 6:46 AM 6harat <bharat.gulati.ce...@gmail.com> wrote: > > > Thanks David for starting this thread. We have also seen this behavior from > > overseer resulting in "orphan collections" or "more than 1 replica created" > > due to timeouts especially when our cluster is scaled up during peak > > traffic days. We call them "orphan collections" too :-) > > While I am still at a nascent stage of my understanding of solr internals, > > I wanted to highlight the below points: (pardon me if these doesn't make > > much sense), > > > > 1. There may be situations where we want solr to still honor the late > > message and hence the functionality needs to be configurable and not a > > default. For instance, during decommissioning of boxes (when we are scaling > > down to our normal cluster size from peak), we send delete replica commands > > for 20+ boxes in a short time frame. Majority of these API hits inevitably > > times out, however we rely upon the behaviour that the cluster after X mins > > is able to reach to the desired state. I'd argue that if you get a timeout telling any system something... all bets are off on what happened and didn't happen. If you change this to use the Solr's async command style, it would be more reliable and wouldn't relate to my proposal. Do note it kind of litters an ID in ZK; it's ideal if your client can tend to deleting the ID but they will be deleted eventually by SolrCloud. > > > > 2. How do we intend to communicate the timeout based rejection of overseer > > message to the end-user I can only answer for "end user" if you mean the client talking to Solr. It would simply get an error response indicating that a timeout occurred. > > 3. In case of fail-over scenario where the overseer leader node goes down > > and is re-elected, the election may have some overhead which may inevitably > > result in many of the piled up messages being rejected due to time > > constraints. Do we intend to pause the clock ticks during this phase or the > > guidance should be to set timeout higher than sum of such possible > > overheads Definitely not pausing the clock. It may be worth repeating what we all know -- in a distributed system, failures (to include timeouts) are going to happen and clients need to be resilient to them (e.g. try again). ~ David > > > > On Wed, Jan 31, 2024 at 11:18 PM David Smiley <dsmi...@apache.org> wrote: > > > > > I have a proposal and am curious what folks think. When the Overseer > > > dequeues an admin command message to process, imagine it being > > > enhanced to examine the "ctime" (creation time) of the ZK message node > > > to determine how long it has been enqueued, and thus roughly how long > > > the client has been waiting. If it's greater than a configured > > > threshold (1 minute?), respond with an error of a timeout nature. > > > "Sorry, the Overseer is so backed up that we fear you have given up; > > > please try again". This would not apply to an "async" style > > > submission. > > > > > > Motivation: Due to miscellaneous reasons at scale that are very user > > > / situation dependent, the Overseer can get seriously backed up. The > > > client, making a typical synchronous call to, say, create a > > > collection, may reach its timeout (say a minute) and has given up. > > > Today, SolrCloud doesn't know this; it goes on its merry way and > > > creates a collection anyway. Depending on how Solr is used, this can > > > be an orphaned collection that the client doesn't want anymore. That > > > is to say, the client wants a collection but it wanted it at the time > > > it asked for it with the name it asked for at that time. If it fails, > > > it will come back later and propose a new name. This doesn't have to > > > be collection creation specific; I'm thinking that in principle it > > > doesn't really matter what the command is. If Solr takes too long for > > > the Overseer to receive the message; just timeout, basically. > > > > > > Thoughts? > > > > > > This wouldn't be a concern for the distributed mode of collection > > > processing as there is no queue bottleneck; the receiving node > > > processes the request immediately. > > > > > > ~ David Smiley > > > Apache Lucene/Solr Search Developer > > > http://www.linkedin.com/in/davidwsmiley > > > > > > --------------------------------------------------------------------- > > > To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org > > > For additional commands, e-mail: dev-h...@solr.apache.org > > > > > > > > > > -- > > Regards > > 6harat > > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@solr.apache.org For additional commands, e-mail: dev-h...@solr.apache.org
