> On May 30, 2017, 10:13 p.m., Stephan Erb wrote: > > Thanks for the patch! That is an interesting idea. I am wondering if it is > > safe in all cases. For example, if I reduce the resource requirements while > > increasing the number of instances, my service could potentially breach his > > assigned quota. > > > > The "sliding update" Bill talks about in > > https://www.youtube.com/watch?v=y1hi7K1lPkk&t=13m25s might be a more robust > > alternative. It will be more complicated to implement though... > > > > I am not working on very large clusters, so I am kind of interested what > > the others have to say here.
Thanks for the comment Stephan! You bring up a good point. >From what I understand, your final configuration cannot exceed your production >quota no matter what. If you reduce the number of resource requirements while >increasing the number of instances, if the overall end resource usage is over >the quota your update will be denied. In the scenario where your end state usage is below your production usage, you may temporarily breach your quota as the update rolls out. For example, if you add new instances and change the resource requirements of all instances, the new instances will be added before the old ones can update. However, at the end of the update, you will be within your quota. This scenario is already occuring with the current ordering (but with killing instances). Right now, we update in a numerical order meaning killing instances happens last. Let's say you increase the resource requirements but reduce the number of instances: at some point during the update you will temporarily exceed your quota as you have updated old instance with more resources (pushing total utilization over the quota) but have yet to kill other instances (bringing utilization back down to end state within quota). - Jordan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59640/#review176371 ----------------------------------------------------------- On May 30, 2017, 9:21 p.m., Jordan Ly wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59640/ > ----------------------------------------------------------- > > (Updated May 30, 2017, 9:21 p.m.) > > > Review request for Aurora, David McLaughlin, Santhosh Kumar Shanmugham, > Stephan Erb, and Zameer Manji. > > > Bugs: AURORA-1928 > https://issues.apache.org/jira/browse/AURORA-1928 > > > Repository: aurora > > > Description > ------- > > Currently, when updating a job we choose to update instances naively by > ascending instance ID number. > However, it would be better to add new instances before killing and updating > older instances. > > This patch makes it so the job updater prefers to create new instances, then > update instances, and finally kill instances. > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/updater/UpdateFactory.java > 14c2d2de3186271819a5f4e527d3b30fd34d2b21 > src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java > 290385d737294e23e9dd50f2631303124aa7af7c > > src/test/java/org/apache/aurora/scheduler/updater/UpdateFactoryImplTest.java > 611f6b8681c8e0b286cd361bdb5ace1fea39d9a5 > > > Diff: https://reviews.apache.org/r/59640/diff/1/ > > > Testing > ------- > > Ran unit tests and integration tests. > > Had to modify some integration tests since we now prefer to create over > update -- needed to change > the ordering of actions. Additionally, some unit tests only specified configs > for one instance even > though desiredInstances is always 2 -- had to make it so the range of > configurations is always 0-2 > when creating. Otherwise, it would try to create instances first even though > the test didn't really > care. > > Tested different update configurations on the Vagrant cluster: only adding > instances, only updating > instances, only killing instances, create & update, update & kill. > > > Thanks, > > Jordan Ly > >
