Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62403 --- @ReviewBot retry - Maxim Khutornenko On Nov. 20, 2014, 3:01 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 20, 2014, 3:01 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java f247ccfb115d4daa10fd4ca46750f708a093791b src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintMatcher.java cc8c5b81baf0cf05d7ac04f69ad2834fa9438aac src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 723e7ab70b8bfd930154dcce385cd1e1d599cf9f src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cf3bb644008274672c77753b94f4d50c99a1a49b src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java d408ec0f7781f65d5b5f215eced5d6255e53b0dd src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d1a70661122802ecfdd8efa2ced567685c08995f src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 168290af9f84750e66dca69cf056dacb5f38aaa3 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62402 --- Ship it! Ship It! - Bill Farner On Nov. 20, 2014, 3:01 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 20, 2014, 3:01 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java f247ccfb115d4daa10fd4ca46750f708a093791b src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintMatcher.java cc8c5b81baf0cf05d7ac04f69ad2834fa9438aac src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 723e7ab70b8bfd930154dcce385cd1e1d599cf9f src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cf3bb644008274672c77753b94f4d50c99a1a49b src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java d408ec0f7781f65d5b5f215eced5d6255e53b0dd src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d1a70661122802ecfdd8efa2ced567685c08995f src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 168290af9f84750e66dca69cf056dacb5f38aaa3 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62405 --- Master (6d06d86) is red with this patch. ./build-support/jenkins/build.sh Installing /home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/lib/python2.7/site-packages/twitter.common.util-0.3.1-py2.7-nspkg.pth Found existing installation: setuptools 3.6 Uninstalling setuptools: Successfully uninstalled setuptools Successfully installed Markdown Pygments ansicolors cov-core coverage lockfile pantsbuild.pants pex psutil py pystache pytest pytest-cov python-daemon requests twitter.common.collections twitter.common.config twitter.common.confluence twitter.common.contextutil twitter.common.decorators twitter.common.dirutil twitter.common.lang twitter.common.log twitter.common.options twitter.common.process twitter.common.string twitter.common.threading twitter.common.util setuptools Cleaning up... Build operating on top level addresses: set([BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD, all)]) Problem executing PythonBuilder for targets OrderedSet([PythonTests(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, api)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, api)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, restarter)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, instance_watcher)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_client)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/au rora/common/auth/BUILD, auth)), PythonThriftLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, transport)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_mux)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, error_handling_thread)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, task_util)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/ BUILD, base)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, http_signaler)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater_util)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, sla)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, common)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, aurora_job_key)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster_option)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, clusters)), PythonLibrary(B uildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-packaged)), PythonThriftLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-test)), PythonThriftLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-storage)), PythonLibrary(BuildFileAddress(/home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater)),
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62407 --- @ReviewBot retry - last attempt - Maxim Khutornenko On Nov. 20, 2014, 3:01 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 20, 2014, 3:01 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java f247ccfb115d4daa10fd4ca46750f708a093791b src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintMatcher.java cc8c5b81baf0cf05d7ac04f69ad2834fa9438aac src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 723e7ab70b8bfd930154dcce385cd1e1d599cf9f src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cf3bb644008274672c77753b94f4d50c99a1a49b src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java d408ec0f7781f65d5b5f215eced5d6255e53b0dd src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d1a70661122802ecfdd8efa2ced567685c08995f src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 168290af9f84750e66dca69cf056dacb5f38aaa3 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62408 --- Master (6d06d86) is red with this patch. ./build-support/jenkins/build.sh File /usr/lib/python2.7/sysconfig.py, line 355, in _init_posix raise IOError(msg) IOError: invalid Python installation: unable to open /x1/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/local/include/python2.7/pyconfig.h (No such file or directory) Build operating on top level addresses: set([BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD, all)]) Problem executing PythonBuilder for targets OrderedSet([PythonTests(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, restarter)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, instance_watcher)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_client)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/au th/BUILD, auth)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, transport)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_mux)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, error_handling_thread)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, task_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, base)), PythonLibrary (BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, http_signaler)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, sla)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, common)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, aurora_job_key)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster_option)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, clusters)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/ workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-packaged)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-test)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-storage)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, job_monitor)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, quota_check)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace /AuroraBot/src/main/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/admin/BUILD, util)),
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62256 --- src/main/java/org/apache/aurora/scheduler/TaskVars.java https://reviews.apache.org/r/27705/#comment104269 This should be the only dynamic one, right? Rack/host limit? Value constraints can be considered static. src/main/java/org/apache/aurora/scheduler/TaskVars.java https://reviews.apache.org/r/27705/#comment104270 I don't have strong data to back this up, but i'm concerned about the performance impact here. In a large/busy cluster, this could be invoked O(100k) times per second. Can you avoid the Set creation? Maybe instead just loop and use two flags? To combat this, we really need to move the nearest fit/miss calculation to the publisher end (which would, unfortunately, break these stats). For now, i think it makes sense to merely be cognizant of the call frequency. - Bill Farner On Nov. 15, 2014, 12:15 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 15, 2014, 12:15 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java 6bfa3ac425ed3045fa60d1b0ca547e9bf3cde37a src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 608903268a0a0d67711bfdc81d2e5b29c335ead2 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 19, 2014, 10:51 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 89 https://reviews.apache.org/r/27705/diff/4/?file=764617#file764617line89 This should be the only dynamic one, right? Rack/host limit? Value constraints can be considered static. Ah, good catch, reodered names to go alphabetically but did not change the values. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62256 --- On Nov. 15, 2014, 12:15 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 15, 2014, 12:15 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java 6bfa3ac425ed3045fa60d1b0ca547e9bf3cde37a src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 608903268a0a0d67711bfdc81d2e5b29c335ead2 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 19, 2014, 10:51 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 252 https://reviews.apache.org/r/27705/diff/4/?file=764617#file764617line252 I don't have strong data to back this up, but i'm concerned about the performance impact here. In a large/busy cluster, this could be invoked O(100k) times per second. Can you avoid the Set creation? Maybe instead just loop and use two flags? To combat this, we really need to move the nearest fit/miss calculation to the publisher end (which would, unfortunately, break these stats). For now, i think it makes sense to merely be cognizant of the call frequency. Refactored to not use any new object creation. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62256 --- On Nov. 15, 2014, 12:15 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 15, 2014, 12:15 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java 6bfa3ac425ed3045fa60d1b0ca547e9bf3cde37a src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 608903268a0a0d67711bfdc81d2e5b29c335ead2 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 20, 2014, 3:01 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- CR comments and rebasing. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java f247ccfb115d4daa10fd4ca46750f708a093791b src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintMatcher.java cc8c5b81baf0cf05d7ac04f69ad2834fa9438aac src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java 723e7ab70b8bfd930154dcce385cd1e1d599cf9f src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cf3bb644008274672c77753b94f4d50c99a1a49b src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java d408ec0f7781f65d5b5f215eced5d6255e53b0dd src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d1a70661122802ecfdd8efa2ced567685c08995f src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 168290af9f84750e66dca69cf056dacb5f38aaa3 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review62323 --- Master (6d06d86) is red with this patch. ./build-support/jenkins/build.sh File /usr/lib/python2.7/sysconfig.py, line 355, in _init_posix raise IOError(msg) IOError: invalid Python installation: unable to open /x1/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/local/include/python2.7/pyconfig.h (No such file or directory) Build operating on top level addresses: set([BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD, all)]) Problem executing PythonBuilder for targets OrderedSet([PythonTests(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, restarter)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, instance_watcher)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_client)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/au th/BUILD, auth)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, transport)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_mux)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, error_handling_thread)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, task_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, base)), PythonLibrary (BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, http_signaler)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, sla)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, common)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, aurora_job_key)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster_option)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, clusters)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/ workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-packaged)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-test)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-storage)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, job_monitor)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, quota_check)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace /AuroraBot/src/main/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/admin/BUILD, util)),
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 14, 2014, 2:24 a.m., Bill Farner wrote:
src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 226
https://reviews.apache.org/r/27705/diff/2/?file=763034#file763034line226
To get the data we want, some extra analysis is needed. Specifically -
if we want to figure out how often a scheduling attempt is vetoed _only_
for static reasons (e.g. insufficient resources), these stats will lack
signal.
Instead, we probably want two counters:
- scheduling_veto_static
- scheduling_veto_dynamic
Does that make sense?
Maxim Khutornenko wrote:
I don't see how more granular data would prevent us from aggregating into
static/dynamic groups. However, having aggregate metrics instead will make it
impossible to do any further analysis when needed. Why not going the more
specific route instead? I would have hard time figuring out what
scheduling_veto_static means without digging through the sources, whereas
something like scheduling_veto_INSUFFICIENT_RESOURCES would immediately
make sense by itself.
Bill Farner wrote:
The problem is that you can't discern when a task didn't match due to
_only_ static reasons. Relevant code in `SchedulingFilterImpl`:
return ImmutableSet.Vetobuilder()
.addAll(getConstraintFilter(attributeAggregate,
attributes).apply(task))
.addAll(getResourceVetoes(offer, task))
.build();
On the other end when you incrmeent counters:
for (Veto veto : event.getVetoes()) {
counters.getUnchecked(vetoStatName(veto)).increment();
}
At this point, you might get vetoes like: `insufficient CPU`,
`insufficient RAM`, `insufficient ports`, `limit not satisfied: host`.
You'll end up with these counter deltas:
`INSUFFICIENT_RESOURCES 3`
`LIMIT_NOT_SATISFIED 1`
As a result, i don't see how we could look at the stats and convince
ourselves which optimization has the greatest payoff, since a single
scheduling round affects multiple counters disproportionately.
Isn't it the same problem with the aggregate counters? I.e. in the above
example we would still see static=1 (or 3?) and dynamic=1.
To address your concern of excessive counting, how about maintaining unique
veto type counters instead? Something like this:
```java
ListMultimapVetoType, Veto index = Multimaps.index(event.getVetoes(),
VETO_TO_VETO_TYPE);
for (VetoType vetoType : index.keys()) {
counters.getUnchecked(vetoStatName(vetoType)).increment();
}
```
For the above example, it would produce:
scheduling_veto_INSUFFICIENT_RESOURCES 1
scheduling_veto_LIMIT_NOT_SATISFIED 1
- Maxim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/#review61385
---
On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/
---
(Updated Nov. 14, 2014, 12:30 a.m.)
Review request for Aurora, Bill Farner and Zameer Manji.
Bugs: AURORA-914
https://issues.apache.org/jira/browse/AURORA-914
Repository: aurora
Description
---
Adding @Timed to trace scheduling latencies and Veto counters per type.
Diffs
-
src/main/java/org/apache/aurora/scheduler/TaskVars.java
cf8f7584afee758c527798914181049051aef0d8
src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java
d2682cd910d248c897e691bcb4c8a3a6f1aec2d2
src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java
e2ba8b8fe978a58d1edcd01963ea020e54529353
src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java
3839083f27ca5d4b93406152559b58b04e912a10
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java
c1c5f26723f1eac3000e09e061b4582f922fded6
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java
cc6b53b3265253f76c1e954c0108aa5936f5cc36
src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java
87203690f09456ac1ca5e9da2b82826d60cbd723
src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java
aaedb3b5ec2cb27550449435efa8f335c6a9baad
src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java
12ea4c67350c2992f59bacd21a99d1413b60b757
src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java
94f0a179b786649775899f855f7c1a0caab7290f
src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java
e113eba1f304279b5ee3d70db1d1ea558efd63ac
src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java
b60b004adbd6753ec6fef125fd70286be5071c56
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 14, 2014, 2:24 a.m., Bill Farner wrote:
src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 226
https://reviews.apache.org/r/27705/diff/2/?file=763034#file763034line226
To get the data we want, some extra analysis is needed. Specifically -
if we want to figure out how often a scheduling attempt is vetoed _only_
for static reasons (e.g. insufficient resources), these stats will lack
signal.
Instead, we probably want two counters:
- scheduling_veto_static
- scheduling_veto_dynamic
Does that make sense?
Maxim Khutornenko wrote:
I don't see how more granular data would prevent us from aggregating into
static/dynamic groups. However, having aggregate metrics instead will make it
impossible to do any further analysis when needed. Why not going the more
specific route instead? I would have hard time figuring out what
scheduling_veto_static means without digging through the sources, whereas
something like scheduling_veto_INSUFFICIENT_RESOURCES would immediately
make sense by itself.
Bill Farner wrote:
The problem is that you can't discern when a task didn't match due to
_only_ static reasons. Relevant code in `SchedulingFilterImpl`:
return ImmutableSet.Vetobuilder()
.addAll(getConstraintFilter(attributeAggregate,
attributes).apply(task))
.addAll(getResourceVetoes(offer, task))
.build();
On the other end when you incrmeent counters:
for (Veto veto : event.getVetoes()) {
counters.getUnchecked(vetoStatName(veto)).increment();
}
At this point, you might get vetoes like: `insufficient CPU`,
`insufficient RAM`, `insufficient ports`, `limit not satisfied: host`.
You'll end up with these counter deltas:
`INSUFFICIENT_RESOURCES 3`
`LIMIT_NOT_SATISFIED 1`
As a result, i don't see how we could look at the stats and convince
ourselves which optimization has the greatest payoff, since a single
scheduling round affects multiple counters disproportionately.
Maxim Khutornenko wrote:
Isn't it the same problem with the aggregate counters? I.e. in the above
example we would still see static=1 (or 3?) and dynamic=1.
To address your concern of excessive counting, how about maintaining
unique veto type counters instead? Something like this:
```java
ListMultimapVetoType, Veto index =
Multimaps.index(event.getVetoes(), VETO_TO_VETO_TYPE);
for (VetoType vetoType : index.keys()) {
counters.getUnchecked(vetoStatName(vetoType)).increment();
}
```
For the above example, it would produce:
scheduling_veto_INSUFFICIENT_RESOURCES 1
scheduling_veto_LIMIT_NOT_SATISFIED 1
Discussed with Bill offline. There is more logic to it. It's not just about
gouping metrics but rather reporting the group when ALL of the Vetos issued
fall into the same group. For example:
- insufficient RAM, limit not satisfied - only static vetos - increment
static counter;
- constraint mismatch, insufficient RAM - mixed static and dynamic vetos -
increment mixed counter;
- constraint mismatch - only dynamic vetos - increment dynamic counter;
- Maxim
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/#review61385
---
On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/
---
(Updated Nov. 14, 2014, 12:30 a.m.)
Review request for Aurora, Bill Farner and Zameer Manji.
Bugs: AURORA-914
https://issues.apache.org/jira/browse/AURORA-914
Repository: aurora
Description
---
Adding @Timed to trace scheduling latencies and Veto counters per type.
Diffs
-
src/main/java/org/apache/aurora/scheduler/TaskVars.java
cf8f7584afee758c527798914181049051aef0d8
src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java
d2682cd910d248c897e691bcb4c8a3a6f1aec2d2
src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java
e2ba8b8fe978a58d1edcd01963ea020e54529353
src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java
3839083f27ca5d4b93406152559b58b04e912a10
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java
c1c5f26723f1eac3000e09e061b4582f922fded6
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java
cc6b53b3265253f76c1e954c0108aa5936f5cc36
src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java
87203690f09456ac1ca5e9da2b82826d60cbd723
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 10:16 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Converted from individual to group veto counters. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61529 --- This patch does not apply cleanly on master (8741cdb), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Nov. 14, 2014, 10:16 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 10:16 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61525 --- src/main/java/org/apache/aurora/scheduler/TaskVars.java https://reviews.apache.org/r/27705/#comment103220 Please define static and dynamic vetos in a comment here. src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java https://reviews.apache.org/r/27705/#comment103227 I think this will fail checkStyle. - Zameer Manji On Nov. 14, 2014, 2:16 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 2:16 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61537 --- Ship it! Ship It! - Zameer Manji On Nov. 14, 2014, 2:16 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 2:16 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 14, 2014, 10:46 p.m., Zameer Manji wrote: src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 64 https://reviews.apache.org/r/27705/diff/3/?file=764035#file764035line64 Please define static and dynamic vetos in a comment here. Thanks for reminding. Meant to do that but it slipped. Done. On Nov. 14, 2014, 10:46 p.m., Zameer Manji wrote: src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java, line 55 https://reviews.apache.org/r/27705/diff/3/?file=764044#file764044line55 I think this will fail checkStyle. Damn intellij. Tweaked a few settings in CodeStyle - Imports to hopefully never see it again. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61525 --- On Nov. 14, 2014, 10:16 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 10:16 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 15, 2014, 12:15 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- CR comments. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java 6bfa3ac425ed3045fa60d1b0ca547e9bf3cde37a src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 608903268a0a0d67711bfdc81d2e5b29c335ead2 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61578 --- Master (8741cdb) is red with this patch. ./build-support/jenkins/build.sh File /usr/lib/python2.7/sysconfig.py, line 355, in _init_posix raise IOError(msg) IOError: invalid Python installation: unable to open /x1/jenkins/jenkins-slave/workspace/AuroraBot/build-support/pants.venv/local/include/python2.7/pyconfig.h (No such file or directory) Build operating on top level addresses: set([BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/BUILD, all)]) Problem executing PythonBuilder for targets OrderedSet([PythonTests(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, api)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, restarter)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, instance_watcher)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_client)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/au th/BUILD, auth)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, transport)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, scheduler_mux)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, error_handling_thread)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, task_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/BUILD, base)), PythonLibrary (BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, http_signaler)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater_util)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, sla)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, common)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, aurora_job_key)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, cluster_option)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, clusters)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/ workspace/AuroraBot/src/main/python/apache/aurora/common/BUILD, shellify)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-packaged)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-test)), PythonThriftLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/thrift/org/apache/aurora/gen/BUILD, py-thrift-storage)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, updater)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, job_monitor)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/client/api/BUILD, quota_check)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace /AuroraBot/src/main/python/apache/aurora/admin/BUILD, host_maintenance)), PythonLibrary(BuildFileAddress(/x1/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/aurora/admin/BUILD, util)),
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61586 --- Ship it! Master (8741cdb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Nov. 15, 2014, 12:15 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 15, 2014, 12:15 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java 6bfa3ac425ed3045fa60d1b0ca547e9bf3cde37a src/main/java/org/apache/aurora/scheduler/events/PubsubEvent.java 4821a7890b77ccb04c10bee6d8b4b9e7216940cc src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 608903268a0a0d67711bfdc81d2e5b29c335ead2 Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 12:30 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- CR comments. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 11, 2014, 10:04 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java, line 360 https://reviews.apache.org/r/27705/diff/1/?file=753241#file753241line360 It's nice when boolean-returning methods have names that self-document what the return value is. In this case, something like `accceptsOffer`. Renamed. On Nov. 11, 2014, 10:04 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java, line 362 https://reviews.apache.org/r/27705/diff/1/?file=753241#file753241line362 odd line break here, don't split up the type parameter. no idea how that happened, fixed. On Nov. 11, 2014, 10:04 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java, line 81 https://reviews.apache.org/r/27705/diff/1/?file=753244#file753244line81 s/rawReason/reasonParameter/? Additionally, this should probably be `String...` to avoid a future yak shave to support non-unary argument vetoes. Renamed. I don't see a use case for String... just yet. Given this is a private constructor it should be easy to add when needed. On Nov. 11, 2014, 10:04 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java, line 283 https://reviews.apache.org/r/27705/diff/1/?file=753245#file753245line283 Can you do this in a SchedulingFilter decorator instead? That way you can hide a bunch of logic from SchedulingFilterImpl. Bill Farner wrote: Oh, looks like the hard work may already be done - can you consume the `Vetoed` pubsub event elsewhere? Sure, moved to TaskVars. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review60861 --- On Nov. 6, 2014, 10:23 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 6, 2014, 10:23 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 1a45d08c8854a335161476c8321c751f763dc12e src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b23457e0e64b490297166131a1b1b51b6d330415 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c37272c9f46c086cb57b79a5202b3bd80e156f07 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 0533baa5e90ca62b8d35ba05474eaa8e27741a5a src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/async/PreemptorImplTest.java c0fa462c0ebe0b06fa354f5f63d5965827c669a1 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0318179cd70661890f5a53908d1985d54474d476 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java bffbf83653535ecd9bf7b149e1e564c5fba56d17 src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java b42f6e2d23eb44605cc84ce01a80f10344a7e43f Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61351 --- Ship it! Master (254e175) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 12:30 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61385 --- src/main/java/org/apache/aurora/scheduler/TaskVars.java https://reviews.apache.org/r/27705/#comment102954 To get the data we want, some extra analysis is needed. Specifically - if we want to figure out how often a scheduling attempt is vetoed _only_ for static reasons (e.g. insufficient resources), these stats will lack signal. Instead, we probably want two counters: - scheduling_veto_static - scheduling_veto_dynamic Does that make sense? - Bill Farner On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 12:30 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 14, 2014, 2:24 a.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 226 https://reviews.apache.org/r/27705/diff/2/?file=763034#file763034line226 To get the data we want, some extra analysis is needed. Specifically - if we want to figure out how often a scheduling attempt is vetoed _only_ for static reasons (e.g. insufficient resources), these stats will lack signal. Instead, we probably want two counters: - scheduling_veto_static - scheduling_veto_dynamic Does that make sense? I don't see how more granular data would prevent us from aggregating into static/dynamic groups. However, having aggregate metrics instead will make it impossible to do any further analysis when needed. Why not going the more specific route instead? I would have hard time figuring out what scheduling_veto_static means without digging through the sources, whereas something like scheduling_veto_INSUFFICIENT_RESOURCES would immediately make sense by itself. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review61385 --- On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 14, 2014, 12:30 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/TaskVars.java cf8f7584afee758c527798914181049051aef0d8 src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java d2682cd910d248c897e691bcb4c8a3a6f1aec2d2 src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java e2ba8b8fe978a58d1edcd01963ea020e54529353 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c1c5f26723f1eac3000e09e061b4582f922fded6 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java cc6b53b3265253f76c1e954c0108aa5936f5cc36 src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 12ea4c67350c2992f59bacd21a99d1413b60b757 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 94f0a179b786649775899f855f7c1a0caab7290f src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java e113eba1f304279b5ee3d70db1d1ea558efd63ac src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 14, 2014, 2:24 a.m., Bill Farner wrote:
src/main/java/org/apache/aurora/scheduler/TaskVars.java, line 226
https://reviews.apache.org/r/27705/diff/2/?file=763034#file763034line226
To get the data we want, some extra analysis is needed. Specifically -
if we want to figure out how often a scheduling attempt is vetoed _only_
for static reasons (e.g. insufficient resources), these stats will lack
signal.
Instead, we probably want two counters:
- scheduling_veto_static
- scheduling_veto_dynamic
Does that make sense?
Maxim Khutornenko wrote:
I don't see how more granular data would prevent us from aggregating into
static/dynamic groups. However, having aggregate metrics instead will make it
impossible to do any further analysis when needed. Why not going the more
specific route instead? I would have hard time figuring out what
scheduling_veto_static means without digging through the sources, whereas
something like scheduling_veto_INSUFFICIENT_RESOURCES would immediately
make sense by itself.
The problem is that you can't discern when a task didn't match due to _only_
static reasons. Relevant code in `SchedulingFilterImpl`:
return ImmutableSet.Vetobuilder()
.addAll(getConstraintFilter(attributeAggregate, attributes).apply(task))
.addAll(getResourceVetoes(offer, task))
.build();
On the other end when you incrmeent counters:
for (Veto veto : event.getVetoes()) {
counters.getUnchecked(vetoStatName(veto)).increment();
}
At this point, you might get vetoes like: `insufficient CPU`, `insufficient
RAM`, `insufficient ports`, `limit not satisfied: host`.
You'll end up with these counter deltas:
`INSUFFICIENT_RESOURCES 3`
`LIMIT_NOT_SATISFIED 1`
As a result, i don't see how we could look at the stats and convince ourselves
which optimization has the greatest payoff, since a single scheduling round
affects multiple counters disproportionately.
- Bill
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/#review61385
---
On Nov. 14, 2014, 12:30 a.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27705/
---
(Updated Nov. 14, 2014, 12:30 a.m.)
Review request for Aurora, Bill Farner and Zameer Manji.
Bugs: AURORA-914
https://issues.apache.org/jira/browse/AURORA-914
Repository: aurora
Description
---
Adding @Timed to trace scheduling latencies and Veto counters per type.
Diffs
-
src/main/java/org/apache/aurora/scheduler/TaskVars.java
cf8f7584afee758c527798914181049051aef0d8
src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java
d2682cd910d248c897e691bcb4c8a3a6f1aec2d2
src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java
e2ba8b8fe978a58d1edcd01963ea020e54529353
src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java
3839083f27ca5d4b93406152559b58b04e912a10
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java
c1c5f26723f1eac3000e09e061b4582f922fded6
src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java
cc6b53b3265253f76c1e954c0108aa5936f5cc36
src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java
87203690f09456ac1ca5e9da2b82826d60cbd723
src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java
aaedb3b5ec2cb27550449435efa8f335c6a9baad
src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java
12ea4c67350c2992f59bacd21a99d1413b60b757
src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java
94f0a179b786649775899f855f7c1a0caab7290f
src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java
e113eba1f304279b5ee3d70db1d1ea558efd63ac
src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java
b60b004adbd6753ec6fef125fd70286be5071c56
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
5c9ea6cf4eb4d99d94f5d61e784dd7c9c480798c
Diff: https://reviews.apache.org/r/27705/diff/
Testing
---
./gradlew -Pq build
Verified new stats in vagrant.
Thanks,
Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
On Nov. 11, 2014, 10:04 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java, line 283 https://reviews.apache.org/r/27705/diff/1/?file=753245#file753245line283 Can you do this in a SchedulingFilter decorator instead? That way you can hide a bunch of logic from SchedulingFilterImpl. Oh, looks like the hard work may already be done - can you consume the `Vetoed` pubsub event elsewhere? - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review60861 --- On Nov. 6, 2014, 10:23 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 6, 2014, 10:23 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 1a45d08c8854a335161476c8321c751f763dc12e src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b23457e0e64b490297166131a1b1b51b6d330415 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c37272c9f46c086cb57b79a5202b3bd80e156f07 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 0533baa5e90ca62b8d35ba05474eaa8e27741a5a src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/async/PreemptorImplTest.java c0fa462c0ebe0b06fa354f5f63d5965827c669a1 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0318179cd70661890f5a53908d1985d54474d476 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java bffbf83653535ecd9bf7b149e1e564c5fba56d17 src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java b42f6e2d23eb44605cc84ce01a80f10344a7e43f Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 1a45d08c8854a335161476c8321c751f763dc12e src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b23457e0e64b490297166131a1b1b51b6d330415 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c37272c9f46c086cb57b79a5202b3bd80e156f07 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 0533baa5e90ca62b8d35ba05474eaa8e27741a5a src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/async/PreemptorImplTest.java c0fa462c0ebe0b06fa354f5f63d5965827c669a1 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0318179cd70661890f5a53908d1985d54474d476 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java bffbf83653535ecd9bf7b149e1e564c5fba56d17 src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java b42f6e2d23eb44605cc84ce01a80f10344a7e43f Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 27705: Adding instrumentation into the scheduling pipeline.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/#review60263 --- Ship it! Master (5250a94) is green with this patch. ./build-support/jenkins/build.sh - Aurora ReviewBot On Nov. 6, 2014, 10:23 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/27705/ --- (Updated Nov. 6, 2014, 10:23 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-914 https://issues.apache.org/jira/browse/AURORA-914 Repository: aurora Description --- Adding @Timed to trace scheduling latencies and Veto counters per type. Diffs - src/main/java/org/apache/aurora/scheduler/async/OfferQueue.java 1a45d08c8854a335161476c8321c751f763dc12e src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b23457e0e64b490297166131a1b1b51b6d330415 src/main/java/org/apache/aurora/scheduler/filter/ConstraintFilter.java 3839083f27ca5d4b93406152559b58b04e912a10 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilter.java c37272c9f46c086cb57b79a5202b3bd80e156f07 src/main/java/org/apache/aurora/scheduler/filter/SchedulingFilterImpl.java 0533baa5e90ca62b8d35ba05474eaa8e27741a5a src/main/java/org/apache/aurora/scheduler/metadata/NearestFit.java 87203690f09456ac1ca5e9da2b82826d60cbd723 src/main/java/org/apache/aurora/scheduler/stats/CachedCounters.java aaedb3b5ec2cb27550449435efa8f335c6a9baad src/test/java/org/apache/aurora/scheduler/async/PreemptorImplTest.java c0fa462c0ebe0b06fa354f5f63d5965827c669a1 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 0318179cd70661890f5a53908d1985d54474d476 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java bffbf83653535ecd9bf7b149e1e564c5fba56d17 src/test/java/org/apache/aurora/scheduler/metadata/NearestFitTest.java b60b004adbd6753ec6fef125fd70286be5071c56 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java b42f6e2d23eb44605cc84ce01a80f10344a7e43f Diff: https://reviews.apache.org/r/27705/diff/ Testing --- ./gradlew -Pq build Verified new stats in vagrant. Thanks, Maxim Khutornenko
