I saw a thread about changing the DB stuff...will that do anything to parallelize this? Would love to help with that effort if possible.
On Mon, Oct 2, 2017 at 6:00 PM, Bill Farner <wfar...@apache.org> wrote: > Or is there a lock somewhere (like DB updates) that makes that useless > > > That is effectively correct. > > On Mon, Oct 2, 2017 at 5:10 PM, Mohit Jaggi <mohit.ja...@uber.com> wrote: > >> True. I also see that the pool of threads for processing offers has a >> size of 1. Am I reading that wrong? Will a larger pool increase >> performance? Or is there a lock somewhere (like DB updates) that makes that >> useless? >> >> On Mon, Oct 2, 2017 at 5:07 PM, Bill Farner <wfar...@apache.org> wrote: >> >>> Is there a place where we store "used" offers >>> >>> >>> Once the scheduler has decided to use an offer, the offer is removed >>> <https://github.com/apache/aurora/blob/6fd6d50288d6155fda212cecab539c5fb669a9aa/src/main/java/org/apache/aurora/scheduler/offers/OfferManager.java#L418> >>> from OfferManager. The situation you describe is indeed possible. >>> However, i don't suspect that there's much to gain from trying to mitigate >>> an accept/rescind race. That type of race also wouldn't cause any state >>> integrity issues in the scheduler, which is what this 'global ban' routine >>> was intending ti plug. >>> >>> On Mon, Oct 2, 2017 at 4:49 PM, Mohit Jaggi <mohit.ja...@uber.com> >>> wrote: >>> >>>> I was wondering if that case can be checked and the banning skipped. Is >>>> there a place where we store "used" offers? At first glance it looks like >>>> there isn't...perhaps deep down in job/task state but that will be too >>>> expensive to check. >>>> >>>> On Mon, Oct 2, 2017 at 4:44 PM, Bill Farner <wfar...@apache.org> wrote: >>>> >>>>> That's true, but it doesn't appear the comment is trying to lay out >>>>> all possible scenarios. Instead, it is attempting to explain the >>>>> rationale >>>>> for offerManager.banOffer(offerId) a few lines later. >>>>> >>>>> On Mon, Oct 2, 2017 at 4:30 PM, Mohit Jaggi <mohit.ja...@uber.com> >>>>> wrote: >>>>> >>>>>> Folks, >>>>>> In the code below, isn't there a 3rd case where the offer was "used" >>>>>> and hence not found for canceling? >>>>>> >>>>>> Mohit. >>>>>> >>>>>> public void handleRescind(OfferID offerId) { >>>>>> log.info("Offer rescinded: {}", offerId.getValue()); >>>>>> >>>>>> // For rescinds, we want to ensure they are processed quickly before >>>>>> we attempt to use an >>>>>> // invalid offer. There are a few scenarios we want to be aware of: >>>>>> // 1. We receive an offer, add it to OfferManager, and then get a >>>>>> rescind. In this scenario, >>>>>> // we can just remove the offer from the offers list. >>>>>> // 2. We receive an offer, but before we add it to the OfferManager >>>>>> list we get a rescind. >>>>>> // In this scenario, we want to ensure that we do not use >>>>>> it/accept it when the executor >>>>>> // finally processes the offer. We will temporarily ban it and >>>>>> add a command for the >>>>>> // executor to unban it so future offers can be processed >>>>>> normally. >>>>>> boolean offerCancelled = offerManager.cancelOffer(offerId); >>>>>> >>>>>> >>>>> >>>> >>> >> >
