> On May 31, 2017, 12:13 a.m., Stephan Erb wrote: > > Thanks for the patch! That is an interesting idea. I am wondering if it is > > safe in all cases. For example, if I reduce the resource requirements while > > increasing the number of instances, my service could potentially breach his > > assigned quota. > > > > The "sliding update" Bill talks about in > > https://www.youtube.com/watch?v=y1hi7K1lPkk&t=13m25s might be a more robust > > alternative. It will be more complicated to implement though... > > > > I am not working on very large clusters, so I am kind of interested what > > the others have to say here. > > Jordan Ly wrote: > Thanks for the comment Stephan! You bring up a good point. > > From what I understand, your final configuration cannot exceed your > production quota no matter what. If you reduce the number of resource > requirements while increasing the number of instances, if the overall end > resource usage is over the quota your update will be denied. > > In the scenario where your end state usage is below your production > usage, you may temporarily breach your quota as the update rolls out. For > example, if you add new instances and change the resource requirements of all > instances, the new instances will be added before the old ones can update. > However, at the end of the update, you will be within your quota. > > This scenario is already occuring with the current ordering (but with > killing instances). Right now, we update in a numerical order meaning killing > instances happens last. Let's say you increase the resource requirements but > reduce the number of instances: at some point during the update you will > temporarily exceed your quota as you have updated old instance with more > resources (pushing total utilization over the quota) but have yet to kill > other instances (bringing utilization back down to end state within quota). > > David McLaughlin wrote: > This is correct. Test coverage for this is here: > https://github.com/apache/aurora/blob/master/src/test/java/org/apache/aurora/scheduler/quota/QuotaManagerImplTest.java#L593
Oh right. Thanks for the clarification! - Stephan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59640/#review176371 ----------------------------------------------------------- On May 30, 2017, 11:21 p.m., Jordan Ly wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59640/ > ----------------------------------------------------------- > > (Updated May 30, 2017, 11:21 p.m.) > > > Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, > Stephan Erb, and Zameer Manji. > > > Bugs: AURORA-1928 > https://issues.apache.org/jira/browse/AURORA-1928 > > > Repository: aurora > > > Description > ------- > > Currently, when updating a job we choose to update instances naively by > ascending instance ID number. > However, it would be better to add new instances before killing and updating > older instances. > > This patch makes it so the job updater prefers to create new instances, then > update instances, and finally kill instances. > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java > 14c2d2de3186271819a5f4e527d3b30fd34d2b21 > src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java > 290385d737294e23e9dd50f2631303124aa7af7c > > src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java > 611f6b8681c8e0b286cd361bdb5ace1fea39d9a5 > > > Diff: https://reviews.apache.org/r/59640/diff/1/ > > > Testing > ------- > > Ran unit tests and integration tests. > > Had to modify some integration tests since we now prefer to create over > update -- needed to change > the ordering of actions. Additionally, some unit tests only specified configs > for one instance even > though desiredInstances is always 2 -- had to make it so the range of > configurations is always 0-2 > when creating. Otherwise, it would try to create instances first even though > the test didn't really > care. > > Tested different update configurations on the Vagrant cluster: only adding > instances, only updating > instances, only killing instances, create & update, update & kill. > > > Thanks, > > Jordan Ly > >
