Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 26, 2015, 8:18 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Changes --- Rebased + fixed merge conflicts. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java dd64a2250ab40663fc7a0296c11206aaed178d7f src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 846914eeba38004e5e7d61ccb76b5219442d8daf src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 65b4836d39f70295264a58e3227863fa4ced705c src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 22faccf42c3cc1181ef2830cd0b86a3146779d14 src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/mesos/TaskExecutors.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora 67d3dbb6ea02baacfbed72e9d70c3109b3a37759 src/test/sh/org/apache/aurora/e2e/validate_serverset.py 66fa965044a4998857b6458582eeb3296558168d Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69666 --- Master (7ba6226) is red with this patch. ./build-support/jenkins/build.sh :assemble :compileJmhJavawarning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.BenchmarkProcessor' less than -source '1.7' 1 warning :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh :jsHint :checkstyleMain :compileTestJava :processTestResources :testClasses :checkstyleTest :findbugsJmh :findbugsMain :findbugsTest :licenseJmh UP-TO-DATE :licenseMain UP-TO-DATE :licenseTest UP-TO-DATE :license UP-TO-DATE :pmdMain :test :jacocoTestReport Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html :analyzeReport Instruction coverage of 0.8913210893561411 exceeds minimum coverage of 0.89. :analyzeReport FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':analyzeReport'. Branch coverage is 0.8343558282208589, but must be greater than 0.835 * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 3 mins 46.842 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 26, 2015, 8:18 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 26, 2015, 8:18 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java dd64a2250ab40663fc7a0296c11206aaed178d7f src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 846914eeba38004e5e7d61ccb76b5219442d8daf src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 65b4836d39f70295264a58e3227863fa4ced705c src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69487 --- Ship it! Ship It! - Zameer Manji On Jan. 23, 2015, 1:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 1:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
On Jan. 23, 2015, 9:44 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/ResourceSlot.java, line 47 https://reviews.apache.org/r/30203/diff/1/?file=831389#file831389line47 Given peak 170MB, rounding to 256 feels too wasteful. This change predates the bug being fixed here, so i would like to avoid attempting to rehash that. On Jan. 23, 2015, 9:44 p.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/configuration/Resources.java, line 206 https://reviews.apache.org/r/30203/diff/1/?file=831394#file831394line206 @VisibleForTesting? Done. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69462 --- On Jan. 23, 2015, 2:06 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 2:06 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 9:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
On Jan. 23, 2015, 10:14 p.m., Zameer Manji wrote: src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java, line 36 https://reviews.apache.org/r/30203/diff/2/?file=832278#file832278line36 Can you add a ticket to track this TODO? https://issues.apache.org/jira/browse/AURORA-1055 - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69479 --- On Jan. 23, 2015, 9:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 23, 2015, 9:48 p.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 24, 2015, 12:33 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Changes --- Needed a few updates to the end-to-end tests to get them to pass. This is needed since the very small tasks used were upsized, and 4 instances (used in the job) would not fit in the slave. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora 67d3dbb6ea02baacfbed72e9d70c3109b3a37759 src/test/sh/org/apache/aurora/e2e/validate_serverset.py 66fa965044a4998857b6458582eeb3296558168d Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/#review69522 --- This patch does not apply cleanly on master (7ba6226), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Jan. 24, 2015, 12:33 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- (Updated Jan. 24, 2015, 12:33 a.m.) Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora 67d3dbb6ea02baacfbed72e9d70c3109b3a37759 src/test/sh/org/apache/aurora/e2e/validate_serverset.py 66fa965044a4998857b6458582eeb3296558168d Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
Review Request 30203: Fix impedance mismatch between offer matching and task launching.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30203/ --- Review request for Aurora, Maxim Khutornenko and Zameer Manji. Bugs: AURORA-1050 https://issues.apache.org/jira/browse/AURORA-1050 Repository: aurora Description --- See linked ticket for context on how this manifested. Please don't be overwhelmed by the large delta in this diff - a majority of it is reorganizing code to live in more appropriate places. This happened because the logic for offer matching and task launching were out of sync. For small tasks (smaller than `MIN_THERMOS_RESOURCES`), an additional amount (`MIN_TASK_RESOURCES`) would be unintentionally added when the task was launched, but this was not considered when comparing the task to offers. The test case `MesosTaskFactoryImplTest.testSmallTaskUpsizing` was added to reproduce this bug. This change does several things to make the situation more sane: - `ResourceSlot` no longer directly accesses command line arguments, to simplify testing - You may no longer create a `ResourceSlot` from its constitutent parts, to prevent accidental misuse - The algorithm used in `ResourceSlot` was simplified such that an epsilon is always used for the executor resources, and that is subtracted from the final resources required by the task. Diffs - src/main/java/org/apache/aurora/scheduler/ResourceSlot.java 0b15834ec67959d3be94f9a5240ed38f43ac4c5b src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 72c7545e7f16549f6a9ccb5fb74a06f154a7ea94 src/main/java/org/apache/aurora/scheduler/async/preemptor/LiveClusterState.java e6bd1b517535cafce4976e585b377065dfd19796 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionVictim.java 024a689d788804e95de76570674b6d4aa77d7495 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 0204d14b19ae412236f19ca274d81decb4eba12d src/main/java/org/apache/aurora/scheduler/configuration/Resources.java 65c4b526c89a4d5607af4424ebe49bb48e296ae9 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c2a342ce07bfb223193886038761f0da5230135d src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 1cb56f19c331508a1585077e9c4a98f52aac343b src/main/java/org/apache/aurora/scheduler/mesos/ExecutorSettings.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 5cc85f1f87f3b8355c89e8ecac19de1122a079e6 src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 7ba946422577c21cbc3b3edf8d30ee313b0ef251 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 5e54364a49a208bd5f19b9649633dc8feca591e9 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 36dbcf73686c5a3ade01f7a10fda8ac4bdbcc7ad src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0b41156f2a574d3d3c2cf840926f307dfb1e726e src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java 265c38d20136210e7639ac8ea915d307a4b72949 src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java c7de6e111300b009e1f9f430624a56100328184e src/test/java/org/apache/aurora/scheduler/mesos/Offers.java PRE-CREATION Diff: https://reviews.apache.org/r/30203/diff/ Testing --- Thanks, Bill Farner
