Re: Review Request 31101: Native Debian packaging for Aurora
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31101/#review78245 --- Ship it! Master (3ab8a9e) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 30, 2015, 7:11 p.m., Benjamin Staffin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31101/ --- (Updated March 30, 2015, 7:11 p.m.) Review request for Aurora and Jake Farrell. Bugs: AURORA-951 https://issues.apache.org/jira/browse/AURORA-951 Repository: aurora Description --- Native Debian packaging for Aurora This uses Debian package dependencies to install gradle and thrift-compiler. Gradle 2.2.1 isn't in Debian or Ubuntu yet, but you can build your own package easily enough using https://github.com/benley/gradle-packaging Diffs - build-support/embed_runner_in_executor.py PRE-CREATION debian/aurora-doc.docs PRE-CREATION debian/aurora-doc.examples PRE-CREATION debian/aurora-executor.dirs PRE-CREATION debian/aurora-executor.install PRE-CREATION debian/aurora-executor.links PRE-CREATION debian/aurora-executor.thermos.default PRE-CREATION debian/aurora-executor.thermos.init PRE-CREATION debian/aurora-executor.thermos.upstart PRE-CREATION debian/aurora-scheduler.default PRE-CREATION debian/aurora-scheduler.init PRE-CREATION debian/aurora-scheduler.install PRE-CREATION debian/aurora-scheduler.links PRE-CREATION debian/aurora-scheduler.postinst PRE-CREATION debian/aurora-scheduler.upstart PRE-CREATION debian/aurora-tools.install PRE-CREATION debian/aurora-tools.links PRE-CREATION debian/changelog PRE-CREATION debian/clusters.json PRE-CREATION debian/compat PRE-CREATION debian/control PRE-CREATION debian/copyright PRE-CREATION debian/pants.ini PRE-CREATION debian/rules PRE-CREATION debian/source/format PRE-CREATION examples/vagrant/aurorabuild.sh 7dc9777c054a8f8a98059f263c49ba2e3e6d7207 Diff: https://reviews.apache.org/r/31101/diff/ Testing --- Built debs in a clean Ubuntu 14.04 environment using git-buildpackage and pbuilder. Have been using debs built from this branch on a testing cluster for a few months now. Thanks, Benjamin Staffin
Re: Review Request 32559: Add Kerberos support to the scheduler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32559/#review78113 --- Ship it! Master (45f2a84) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 27, 2015, 8:50 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32559/ --- (Updated March 27, 2015, 8:50 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-812 https://issues.apache.org/jira/browse/AURORA-812 Repository: aurora Description --- Support authenticating to the scheduler API with Kerberos. Diffs - config/findbugs/excludeFilter.xml 5ff5f87d6bb7d8bf3d9ecb1bc6c6b1a524d1bf15 examples/vagrant/provision-dev-cluster.sh ae500436e703140065e5c16fc0e38dbe3214e69f examples/vagrant/upstart/aurora-scheduler-kerberos.conf PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java ec6a02c4086ee0d5a7529083030d978ea889f677 src/main/java/org/apache/aurora/scheduler/http/api/security/AuthorizeHeaderToken.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/IniShiroRealmModule.java 58b559e96402ef282dc0e5ac2de0930906256487 src/main/java/org/apache/aurora/scheduler/http/api/security/Kerberos5Realm.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModule.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/KerberosPrincipalParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilter.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroUtils.java PRE-CREATION src/main/python/apache/aurora/client/hooks/BUILD 49d73dfddbcee97ddde1a483f6697865d2566508 src/main/resources/org/apache/aurora/scheduler/http/api/security/jaas.conf.st PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/AuthorizeHeaderTokenTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModuleTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/KerberosPrincipalParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilterTest.java PRE-CREATION src/test/python/apache/aurora/client/hooks/BUILD 43f59bc5a2397ad851211e064d83ef6e6de81972 src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh PRE-CREATION Diff: https://reviews.apache.org/r/32559/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/test_kerberos_end_to_end.sh Thanks, Kevin Sweeney
Re: Review Request 32559: Add Kerberos support to the scheduler
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32559/#review78006 --- Ship it! Master (45f2a84) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 27, 2015, 1:23 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32559/ --- (Updated March 27, 2015, 1:23 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-812 https://issues.apache.org/jira/browse/AURORA-812 Repository: aurora Description --- Support authenticating to the scheduler API with Kerberos. Diffs - config/findbugs/excludeFilter.xml 5ff5f87d6bb7d8bf3d9ecb1bc6c6b1a524d1bf15 examples/vagrant/provision-dev-cluster.sh ae500436e703140065e5c16fc0e38dbe3214e69f examples/vagrant/upstart/aurora-scheduler-kerberos.conf PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java ec6a02c4086ee0d5a7529083030d978ea889f677 src/main/java/org/apache/aurora/scheduler/http/api/security/AuthorizeHeaderToken.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/IniShiroRealmModule.java 58b559e96402ef282dc0e5ac2de0930906256487 src/main/java/org/apache/aurora/scheduler/http/api/security/Kerberos5Realm.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModule.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/KerberosPrincipalParser.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilter.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroUtils.java PRE-CREATION src/main/resources/org/apache/aurora/scheduler/http/api/security/jaas.conf.st PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/AuthorizeHeaderTokenTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/Kerberos5ShiroRealmModuleTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/KerberosPrincipalParserTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilterTest.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh PRE-CREATION Diff: https://reviews.apache.org/r/32559/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/test_kerberos_end_to_end.sh Thanks, Kevin Sweeney
Re: Review Request 32329: Extract job key from RPC parameters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32329/#review78142 --- Master (3ab8a9e) is red with this patch. ./build-support/jenkins/build.sh :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 4 mins 2.721 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1)) Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2)) pip can't proceed with requirement 'CherryPy==3.6.0 (from -r /dev/fd/63 (line 2))' due to a pre-existing build directory. location: /tmp/user/2395/pip_build_jenkins/CherryPy This is likely due to a previous installation that failed. pip is being responsible and not assuming it can delete this. Please delete it and try again. Storing debug log for failure in /home/jenkins/.pip/pip.log I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 28, 2015, 12:02 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32329/ --- (Updated March 28, 2015, 12:02 a.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1187 https://issues.apache.org/jira/browse/AURORA-1187 Repository: aurora Description --- Apologies for the large diff, this wound up needing to input validation at the AOP layer. Probably the best place to start reading this diff is ApiSecurityIT to see the feature this patch enables. Diffs - config/pmd/custom.xml 521fd500146eb2e45f8e77c5c3c0cce330fedabb src/main/java/org/apache/aurora/scheduler/http/api/ApiBeta.java 827e85b6cac8bd52359610bbc2002973a769705c src/main/java/org/apache/aurora/scheduler/http/api/ApiModule.java 2408cd1f9af5f109a339f5c78134465cb117f7fc src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java ec6a02c4086ee0d5a7529083030d978ea889f677 src/main/java/org/apache/aurora/scheduler/http/api/security/AuthorizingParam.java 808987939b2c4a850e488dc033b50b0178e95ba0 src/main/java/org/apache/aurora/scheduler/http/api/security/FieldGetter.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/FieldGetters.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthenticatingThriftInterceptor.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptor.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroThriftInterceptor.java 4e341e05c34b1be38f0040c26b671a0cc797a771 src/main/java/org/apache/aurora/scheduler/http/api/security/ThriftFieldGetter.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 5588d1793d6713ee4581ac9f938d9a8689acb315 src/main/java/org/apache/aurora/scheduler/thrift/aop/AopModule.java bdd2185f3a7a94b39bcec3c73455e970d87f0c6a src/test/java/org/apache/aurora/scheduler/http/api/ApiBetaTest.java cafd10f6b705568588c1b92644b482003242fe2e src/test/java/org/apache/aurora/scheduler/http/api/ApiIT.java ed284f46ac8f01bd6d9e317f995f16d6e666a68d src/test/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityIT.java 76cb691e6d7d4fada3a18fde73aceed7039bcaa4 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthenticatingThriftInterceptorTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptorTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroThriftInterceptorTest.java d2ba2730c4509dc9a636fd32e9244b0d7fa2884f src/test/java/org/apache/aurora/scheduler/http/api/security/ThriftFieldGetterTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 1f24e7d47e1f777ffef19a73d01171fcacd31cdb src/test/java/org/apache/aurora/scheduler/thrift/aop/AopModuleTest.java
Re: Review Request 32597: Improving async preemptor efficiency.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/#review78150 --- This patch does not apply cleanly on master (3ab8a9e), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On March 28, 2015, 3:32 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/ --- (Updated March 28, 2015, 3:32 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- This is addressing the preemption efficiency loss due to making preemptor async. Summary: - Implemented fair task processing by evaluating pending tasks in the order of their TaskGroupKeys (largest count first) - Inverted the traversal of pending tasks to better reuse loop invariants. Every slave is sized up against all pending tasks until a match is found - Moved relevant functionality from PreemptionSlotFinder (now PreemptionVictimFilter) into PendingTaskProcessor. The bulk of new functionality is in PendingTaskIterator and PendingTaskProcessor. The rest is refactoring to adjust to the new traversal approach. Will not apply cleanly. Diffed against https://reviews.apache.org/r/32352/ Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskIterator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlot.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java 4ca36e5fe2cfd326f4d4f37f70dbcd0060109e73 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotFinder.java 84bcdc57d798aca229d4f184cae065ec4dcf8fc5 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 84791a272f245c729706af95d70c7f1631bfe99c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 7034a07eae1f94d7a0bbccdf8146cf3ed0a5424e src/main/java/org/apache/aurora/scheduler/base/TaskGroupKey.java 6af3949b85297043640edccc1a490906c0fcff6c src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java d212bbdf3c37be8e1e00eb169cf40b90fe69ed5f src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskIteratorTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java d17c4fb513afdb7d8ef6d7c2b0aef86c1f47c082 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java b80e558f18b817814e4768b13ff94aa816d28543 Diff: https://reviews.apache.org/r/32597/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant Thanks, Maxim Khutornenko
Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/#review79809 --- Master (eb9e408) is red with this patch. ./build-support/jenkins/build.sh :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 4 mins 7.27 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1)) Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2)) pip can't proceed with requirement 'CherryPy==3.6.0 (from -r /dev/fd/63 (line 2))' due to a pre-existing build directory. location: /tmp/user/2395/pip_build_jenkins/CherryPy This is likely due to a previous installation that failed. pip is being responsible and not assuming it can delete this. Please delete it and try again. Storing debug log for failure in /home/jenkins/.pip/pip.log I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 11, 2015, 6:22 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/ --- (Updated April 11, 2015, 6:22 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- This allows the store implementation to stream if it wishes, which we might want to take advantage of when the store is backed by a database. One thing worth noting is that this change is a good example of why using wide types at call sites is important. There were many callers of TaskStore.fetchTasks that were already assigning as `Iterable`, and they were immune to this refactor. Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java ce87344dc18818faa7a1a0298143dc81fff7 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 3b5dcf819228297b3840cb01788c3085759f3c4a src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java ed82ae99f23d5a7f1634261205cbe5339fe4cec8 src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 09be4dc8d6318ccfdf10397585ef7989221afd7f src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java b6a7b4aca841025c790f4b282f0490ff5bb23b2f src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java a8e3b14b3f8147f60871f9dfa84616425636e10a src/main/java/org/apache/aurora/scheduler/storage/Storage.java 6180a36255253542197234ec610eea01366135bc src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java b76c937281971a505db4b352b80cbb7ccfcafcca src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 38764e5e7484e2fe152460ac7920be9a5799b85d src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 2102adb3844fc1501ff9d672a2db78a69513231b src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 7aef1ca786fea521256f69f6a9e7a978649a7195 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java b7d3874e9f597d93c9f623773a3a0289c2b76d6d src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java c5643d9d99fc46d55fd6c48161230139fb7f12b8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 88c0163170ebc25995d9ef8b1543335a4322bb8e src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java bcd1b4e854f5ea227268c73156bc97c7c937c1de src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 281f4e02650727aa5d0a35a09dcf0eb823ad1b50 src/test/java/org
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/#review79976 --- Ship it! Master (9aab87f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 1:09 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 1:09 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java c5643d9d99fc46d55fd6c48161230139fb7f12b8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 88c0163170ebc25995d9ef8b1543335a4322bb8e src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java bcd1b4e854f5ea227268c73156bc97c7c937c1de src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 281f4e02650727aa5d0a35a09dcf0eb823ad1b50 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33106: Simplify AttributeAggregate.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33106/#review79986 --- Ship it! Master (9aab87f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 4 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33106/ --- (Updated April 14, 2015, 4 a.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- Use `Multiset` instead of `AtomicLongMap`, introduce a convenience constant for unit tests to use. Diffs - src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java ce87344dc18818faa7a1a0298143dc81fff7 src/main/java/org/apache/aurora/scheduler/filter/AttributeAggregate.java ed82ae99f23d5a7f1634261205cbe5339fe4cec8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java c5643d9d99fc46d55fd6c48161230139fb7f12b8 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 88c0163170ebc25995d9ef8b1543335a4322bb8e src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java bcd1b4e854f5ea227268c73156bc97c7c937c1de src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 281f4e02650727aa5d0a35a09dcf0eb823ad1b50 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModuleTest.java 7e2d1c54362b33cc3507a4bc3e3ccc02ca29bd6f src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java b80e558f18b817814e4768b13ff94aa816d28543 src/test/java/org/apache/aurora/scheduler/events/NotifyingSchedulingFilterTest.java 61cea326ababcd6242a3c5a6dcf8d0b3ca7fbdd6 src/test/java/org/apache/aurora/scheduler/filter/AttributeAggregateTest.java 4b565769d79862326efcb31be694f95f333c89c6 src/test/java/org/apache/aurora/scheduler/filter/SchedulingFilterImplTest.java d06b89cc319aa7b4479124cbb2cb224cdb662e05 src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java aca0234e037e85202d182affa2c0e988c6cfc854 Diff: https://reviews.apache.org/r/33106/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33366: Fix the path to download stdout/stderr
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33366/#review80823 --- Ship it! Master (b18dc44) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 20, 2015, 9 p.m., Bhuvan Arumugam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33366/ --- (Updated April 20, 2015, 9 p.m.) Review request for Aurora. Bugs: AURORA-1135 https://issues.apache.org/jira/browse/AURORA-1135 Repository: aurora Description --- Download url for stdout/stderr broken Make it relative to sandbox. Testing Done: ./pants test.pytest --no-fast --options='--verbose' src/test/python:: Bugs closed: AURORA-1135 Diffs - src/main/python/apache/thermos/observer/http/file_browser.py 2f48594a87724b84bad7e149eb4d63c71ea954ee src/test/python/apache/thermos/observer/http/BUILD PRE-CREATION src/test/python/apache/thermos/observer/http/test_file_browser.py PRE-CREATION Diff: https://reviews.apache.org/r/33366/diff/ Testing --- Thanks, Bhuvan Arumugam
Re: Review Request 33374: Resuming blocked updates on restart.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33374/#review80852 --- Ship it! Master (4b90339) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 20, 2015, 10:19 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33374/ --- (Updated April 20, 2015, 10:19 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1285 https://issues.apache.org/jira/browse/AURORA-1285 Repository: aurora Description --- Changing the `systemResume()` to properly re-initialize `PulseState` for all active coordinated updates. Diffs - src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java ac15217fa9eb63e7292d23702e37c6fd194fbaed src/main/java/org/apache/aurora/scheduler/updater/JobUpdateStateMachine.java 74e915ce678855aa2a2ae44ff59cfba259447796 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java dd4c94097161477e8161fb7e440e4c41e2a61f21 Diff: https://reviews.apache.org/r/33374/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 32597: Improving async preemptor efficiency.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/#review80873 --- Ship it! Master (4b90339) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 21, 2015, 1:12 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/ --- (Updated April 21, 2015, 1:12 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- This is addressing the preemption efficiency loss due to making preemptor async. Summary: - Implemented fair task processing by evaluating pending task groups in round-robin fashion (e.g.: {G1:3, G2:2} - {G1, G2, G1, G2, G1}) against all available slaves. - Moved relevant functionality from PreemptionSlotFinder (now PreemptionVictimFilter) into PendingTaskProcessor. The bulk of new functionality is in PendingTaskIterator and PendingTaskProcessor. The rest is refactoring to adjust to the new traversal approach. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 00919b7910704c5025465e1071378a978e5e60a3 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionProposal.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotFinder.java f16f964f56f0f9da523950891293083f1bd86780 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 5200811ec8fe6fedf42ae2712f29051a8c0bf4a9 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorMetrics.java dc7eb4421ff305dca32f36c83605c2864fea8b11 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 7cea881a8c3c11142bd04b3c794cd86a310b15e7 src/main/java/org/apache/aurora/scheduler/base/TaskGroupKey.java 6af3949b85297043640edccc1a490906c0fcff6c src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 7bb1e7a9f27088636d6549c089b1d079dfeaf2ee src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 8a9a3b7d9686e29632f4e267f591cdb19826e0e7 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 64283fab8c61b841007d7c0a02b083b3067bc78d src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java eed2de99a145dd2124b7f2b4d401214f1d8adf2e Diff: https://reviews.apache.org/r/32597/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant Thanks, Maxim Khutornenko
Re: Review Request 33366: Fix the path to download stdout/stderr
/test/python/apache/aurora/client/api/test_task_util.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/__init__.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/client/api/test_restarter.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/config/test_thrift.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/config/test_base.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/config/test_loader.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/python/apache/aurora/config/test_constraint_parsing.py Everything Looks Good! I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 20, 2015, 8:15 p.m., Bhuvan Arumugam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33366/ --- (Updated April 20, 2015, 8:15 p.m.) Review request for Aurora. Bugs: AURORA-1135 https://issues.apache.org/jira/browse/AURORA-1135 Repository: aurora Description --- Download url for stdout/stderr broken Make it relative to sandbox. Testing Done: ./pants test.pytest --no-fast --options='--verbose' src/test/python:: Bugs closed: AURORA-1135 Diffs - src/main/python/apache/thermos/observer/http/file_browser.py 2f48594a87724b84bad7e149eb4d63c71ea954ee src/test/python/apache/thermos/observer/http/BUILD PRE-CREATION src/test/python/apache/thermos/observer/http/test_file_browser.py PRE-CREATION Diff: https://reviews.apache.org/r/33366/diff/ Testing --- Thanks, Bhuvan Arumugam
Re: Review Request 33411: Add typed Shiro permissions SPI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33411/#review81088 --- Ship it! Master (8ba1b11) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 21, 2015, 11 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33411/ --- (Updated April 21, 2015, 11 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Repository: aurora Description --- Working on an implementation of a Shiro Realm to replace the old CapabilityValidator I realized we're missing some information with the new API. This patch allows a Realm implementation to optionally introspect a permission check for Aurora-specific information about exactly what's being attempted, while maintaining compatibility with Shiro realms that don't know anything about Aurora, such as the included IniRealm. To do this I've added a new SPI package, and documented backwards-compatibility considerations with it. The idea is that a third party can write an Aurora-aware Shiro Realm module against a stable ABI and gets one release to update it if we want to make changes to it. Diffs - config/findbugs/excludeFilter.xml 0bff71c33dff8c92fdf5e841c04ee1460c50937b src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java 0265e2a1e4cad3f569501521f1bf25d7caa9da44 src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingInterceptor.java 7a124cccad2e02e7c4f16c3c7fc8bd662bcb5360 src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptor.java fde6c84e297d35b3e10f12a95e12dfb7ab1b477f src/main/java/org/apache/aurora/scheduler/spi/Permissions.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/spi/package-info.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingInterceptorTest.java 16f2da5207ce2a3f866fa3a51c02c11d1b58a439 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptorTest.java 781cf5add1555187757a0254a96f201bc74a1e27 Diff: https://reviews.apache.org/r/33411/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 32597: Improving async preemptor efficiency.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/#review81085 --- Ship it! Master (d10d2d1) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 21, 2015, 9:15 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/ --- (Updated April 21, 2015, 9:15 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- This is addressing the preemption efficiency loss due to making preemptor async. Summary: - Implemented fair task processing by evaluating pending task groups in round-robin fashion (e.g.: {G1:3, G2:2} - {G1, G2, G1, G2, G1}) against all available slaves. - Moved relevant functionality from PreemptionSlotFinder (now PreemptionVictimFilter) into PendingTaskProcessor. The bulk of new functionality is in PendingTaskIterator and PendingTaskProcessor. The rest is refactoring to adjust to the new traversal approach. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 00919b7910704c5025465e1071378a978e5e60a3 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionProposal.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotFinder.java f16f964f56f0f9da523950891293083f1bd86780 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 5200811ec8fe6fedf42ae2712f29051a8c0bf4a9 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorMetrics.java dc7eb4421ff305dca32f36c83605c2864fea8b11 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 7cea881a8c3c11142bd04b3c794cd86a310b15e7 src/main/java/org/apache/aurora/scheduler/base/TaskGroupKey.java 6af3949b85297043640edccc1a490906c0fcff6c src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 7bb1e7a9f27088636d6549c089b1d079dfeaf2ee src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 8a9a3b7d9686e29632f4e267f591cdb19826e0e7 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 64283fab8c61b841007d7c0a02b083b3067bc78d src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java eed2de99a145dd2124b7f2b4d401214f1d8adf2e Diff: https://reviews.apache.org/r/32597/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant Thanks, Maxim Khutornenko
Re: Review Request 33537: Document Aurora security features.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/#review81565 --- Ship it! Master (94fe6c9) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 24, 2015, 10:40 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/ --- (Updated April 24, 2015, 10:40 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-817 https://issues.apache.org/jira/browse/AURORA-817 Repository: aurora Description --- SSIA A rendered version is available on github as https://github.com/kevints/aurora/blob/kts/security-docs/docs/security.md - feel free to send pull requests to fix typos, etc, from the markdown editor there and I will incorporate them into this review. Diffs - docs/security.md PRE-CREATION Diff: https://reviews.apache.org/r/33537/diff/ Testing --- Thanks, Kevin Sweeney
Re: Review Request 33456: Adding logging threadpool executor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33456/#review81240 --- Ship it! Master (352e0ef) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 22, 2015, 10:58 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33456/ --- (Updated April 22, 2015, 10:58 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Our async EventBus is using a regular executor thus potentially hiding unhandled errors. Diffs - src/main/java/org/apache/aurora/scheduler/base/AsyncUtil.java f657e057b5bbff69971876e104ff0e47b2dc4faa src/main/java/org/apache/aurora/scheduler/events/PubsubEventModule.java 3a4d40adc1abe170b5b80644db9f079751d8a9bf src/test/java/org/apache/aurora/scheduler/base/AsyncUtilTest.java e990f528aac768b5c9b829c9544045a831e094fe Diff: https://reviews.apache.org/r/33456/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 33366: download url for stdout/stderr broken
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33366/#review80771 --- Master (b18dc44) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 20, 2015, 5:57 p.m., Bhuvan Arumugam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33366/ --- (Updated April 20, 2015, 5:57 p.m.) Review request for Aurora. Bugs: AURORA-1135 https://issues.apache.org/jira/browse/AURORA-1135 Repository: aurora Description --- Fix the path to download stdout/stderr. Make it relative to sandbox. Testing Done: ./pants test.pytest --no-fast --options='--verbose' src/test/python:: Bugs closed: AURORA-1135 Diffs - src/main/python/apache/thermos/observer/http/file_browser.py 2f48594a87724b84bad7e149eb4d63c71ea954ee Diff: https://reviews.apache.org/r/33366/diff/ Testing --- Thanks, Bhuvan Arumugam
Re: Review Request 33677: Remove dead code related to Java executor.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33677/#review81968 --- Master (e311dbe) is red with this patch. ./build-support/jenkins/build.sh src.test.python.apache.aurora.client.cli.plugins . SUCCESS src.test.python.apache.aurora.client.cli.quota . SUCCESS src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS src.test.python.apache.aurora.common.test_aurora_job_key . SUCCESS src.test.python.apache.aurora.common.test_cluster . SUCCESS src.test.python.apache.aurora.common.test_cluster_option . SUCCESS src.test.python.apache.aurora.common.test_clusters . SUCCESS src.test.python.apache.aurora.common.test_http_signaler . SUCCESS src.test.python.apache.aurora.common.test_pex_version . SUCCESS src.test.python.apache.aurora.common.test_shellify . SUCCESS src.test.python.apache.aurora.common.test_transport . SUCCESS src.test.python.apache.aurora.config.test_base . SUCCESS src.test.python.apache.aurora.config.test_constraint_parsing . SUCCESS src.test.python.apache.aurora.config.test_loader . SUCCESS src.test.python.apache.aurora.config.test_thrift . SUCCESS src.test.python.apache.aurora.executor.common.announcer . FAILURE src.test.python.apache.aurora.executor.common.path_detector . SUCCESS src.test.python.apache.aurora.executor.common.task_info . SUCCESS src.test.python.apache.aurora.executor.executor_base . SUCCESS src.test.python.apache.aurora.executor.executor_vars . SUCCESS src.test.python.apache.aurora.executor.status_manager . SUCCESS src.test.python.apache.aurora.executor.thermos_task_runner . SUCCESS src.test.python.apache.thermos.cli.commands.commands . SUCCESS src.test.python.apache.thermos.cli.common . SUCCESS src.test.python.apache.thermos.cli.main . SUCCESS src.test.python.apache.thermos.common.test_pathspec . SUCCESS src.test.python.apache.thermos.core.test_runner_integration . SUCCESS src.test.python.apache.thermos.monitoring.test_disk . SUCCESS FAILURE [31m FAILURE[0m I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 29, 2015, 3:25 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33677/ --- (Updated April 29, 2015, 3:25 p.m.) Review request for Aurora and Brian Wickman. Bugs: AURORA-195
Re: Review Request 33537: Document Aurora security features.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/#review81751 --- Ship it! Master (297c0eb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 27, 2015, 11:24 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/ --- (Updated April 27, 2015, 11:24 p.m.) Review request for Aurora, David McLaughlin and Bill Farner. Bugs: AURORA-817 https://issues.apache.org/jira/browse/AURORA-817 Repository: aurora Description --- SSIA A rendered version is available on github as https://github.com/kevints/aurora/blob/kts/security-docs/docs/security.md - feel free to send pull requests to fix typos, etc, from the markdown editor there and I will incorporate them into this review. Diffs - docs/deploying-aurora-scheduler.md 70d6e3d5a74cab808b327574c79f5b794aa63c28 docs/security.md PRE-CREATION Diff: https://reviews.apache.org/r/33537/diff/ Testing --- Thanks, Kevin Sweeney
Re: Review Request 33608: Added a status update throughput benchmark.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/#review81765 --- Master (e9d723d) is red with this patch. ./build-support/jenkins/build.sh :compileJavaNote: Writing file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2 :processResources :classes :jar :startScripts :distTar :distZip :assemble :compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh[ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:1: Line does not match expected header line of '^\/\*\*$'. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:68: Wrong order for 'com.twitter.common.application.ShutdownStage' import. Order should be: java, javax, scala, com, net, org. Each group should be separated by a single blank line. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:109: Dont System.exit(), throw a RuntimeException() [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:121: Dont System.exit(), throw a RuntimeException() [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:134: Dont System.exit(), throw a RuntimeException() [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeCommand.java:1: Missing a header - not enough lines in file. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java:1: Line does not match expected header line of '^\/\*\*$'. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java:45: Line is longer than 100 characters (found 132). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java:50: Line is longer than 100 characters (found 108). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java:55: Line is longer than 100 characters (found 119). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java:90: Line is longer than 100 characters (found 112). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeUncaughtExceptionHandler.java:1: Missing a header - not enough lines in file. FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':checkstyleJmh'. Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/jmh.xml * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 1 mins 42.956 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 28, 2015, 12:57 a.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/ --- (Updated April 28, 2015, 12:57 a.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1283 https://issues.apache.org/jira/browse/AURORA-1283 Repository: aurora Description --- In order to justify doing asynchronous batch acknowledgements and to better understand status update throughput, this introduces a benchmark. Note that this assumes that status update processing is not synchronous, so that the benchmark doesn't need to be updated for AURORA-1228. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeCommand.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 45de15a57baf7a2f7d437b590935714e28777f35 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark
Re: Review Request 33243: Use explicit status update acknowledgements.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/#review81747 --- Ship it! Master (94fe6c9) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 27, 2015, 10:44 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/ --- (Updated April 27, 2015, 10:44 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- The scheduler now explicitly acknowledges updates. I left the structure as is, per Bill's suggestion. Diffs - src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java db7aa74c2dfca3d520d978d7837d1f2639e6c05c src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java f40746b5712f21e551c06eeb1aa379ebdcdbc693 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java 4ef23ea6ebd2c9587a8356507fcb49a36b9de219 Diff: https://reviews.apache.org/r/33243/diff/ Testing --- Ran the unit tests and the end-to-end test. Thanks, Ben Mahler
Re: Review Request 33243: Use explicit status update acknowledgements.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/#review81750 --- Ship it! Master (297c0eb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 27, 2015, 11:14 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/ --- (Updated April 27, 2015, 11:14 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- The scheduler now explicitly acknowledges updates. I left the structure as is, per Bill's suggestion. Diffs - src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java db7aa74c2dfca3d520d978d7837d1f2639e6c05c src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java f40746b5712f21e551c06eeb1aa379ebdcdbc693 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java 4ef23ea6ebd2c9587a8356507fcb49a36b9de219 Diff: https://reviews.apache.org/r/33243/diff/ Testing --- Ran the unit tests and the end-to-end test. Thanks, Ben Mahler
Re: Review Request 33783: Add stats for update state transitions.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33783/#review82322 --- Ship it! Master (35c511e) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 2, 2015, 6:41 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33783/ --- (Updated May 2, 2015, 6:41 p.m.) Review request for Aurora and David McLaughlin. Bugs: AURORA-1259 https://issues.apache.org/jira/browse/AURORA-1259 Repository: aurora Description --- This required moving `FakeStatsProvider` from src/test to src/main, since it's now used by `DbUtil`. I'm not totally happy about that, but i didn't see a more favorable approach. Diffs - config/legacy_untested_classes.txt 4d090a8fc1344531772e1b7025d77c2ff0a30035 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java 94ce5c3499ced1b63abf19787acc21b2cd4d0c75 src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java a10a9e79c2e294d1f701e36a522e34222db1c3a3 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 219676fab301542c129794416630070921225439 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java f18619a27eeb2aea8dcf01e54c23ed7d1c7d3d87 src/test/java/org/apache/aurora/scheduler/testing/FakeStatsProvider.java Diff: https://reviews.apache.org/r/33783/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33705: Don't retry API requests that fail with auth errors.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33705/#review82071 --- Ship it! Master (f77daf7) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 30, 2015, 12:12 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33705/ --- (Updated April 30, 2015, 12:12 a.m.) Review request for Aurora, Joshua Cohen and Kevin Sweeney. Bugs: AURORA-1248 https://issues.apache.org/jira/browse/AURORA-1248 Repository: aurora Description --- The only thing i really don't care for in this patch is that `add_auth_error_handler` alters a 'private' field, but i was unable to come up with an alternative that wouldn't seem to spiral into a refactor. Advice/opinions solicited. Diffs - src/main/python/apache/aurora/client/api/scheduler_client.py 3f9c691056ee50103a7857bea43160c75388f2aa src/main/python/apache/aurora/client/cli/__init__.py 10d943215282a83ebe0cadac1592848d797c3e79 src/main/python/apache/aurora/client/cli/context.py adbffc4aa07c242691e87e0fcfc9f5b840999d1d src/main/python/apache/aurora/common/transport.py eefe8f7b50e0e0040e28ff47b226e91316f1911e src/test/python/apache/aurora/client/api/test_scheduler_client.py 9319728d3a95b158cb734f4cbc6be97187d43def src/test/python/apache/aurora/client/cli/test_context.py d63f060ea6cf747792f27499da901dd7cecae596 src/test/python/apache/aurora/common/test_transport.py e4f93ef9dd6bf0d54166bdf6afb247fad871ea10 src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh cc7cdee42b7c03dd1f121963129d94e67cb9e2a2 Diff: https://reviews.apache.org/r/33705/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33608: Added a status update throughput benchmark.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/#review82077 --- Master (f77daf7) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 30, 2015, 12:35 a.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/ --- (Updated April 30, 2015, 12:35 a.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1283 https://issues.apache.org/jira/browse/AURORA-1283 Repository: aurora Description --- In order to justify doing asynchronous batch acknowledgements and to better understand status update throughput, this introduces a benchmark. Note that this assumes that status update processing is not synchronous, so that the benchmark doesn't need to be updated for AURORA-1228. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java c54619f7cd617b48069363173dcf63b6254b4095 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java d7d659bb13f085ff06291ef0033572f8bdf29874 Diff: https://reviews.apache.org/r/33608/diff/ Testing --- Ran the benchmarks against the existing code and some pending code I have for AURORA-1228 to demonstrate the improvement. The end to end tests are broken, appears to be unrelated to my change from what I can tell. Thanks, Ben Mahler
Re: Review Request 33611: Add benchmarks for fetching tasks over the API.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33611/#review81853 --- Ship it! Master (e9d723d) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 28, 2015, 6:16 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33611/ --- (Updated April 28, 2015, 6:16 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- Add a benchmark for unscoped API task queries. This change also makes it possible to run specific benchmarks. I'm also sneaking in a cleanup - removing unnecessary parameter to `DbModule.testModule()`. Diffs - build.gradle 470d11ee0ab9f21e92d3796ab29e038cd9cce0dc src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 6ec0e14355850c8859f675d8fb2bc56fb64cb8b8 src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java a4abbd8129c5e4799a07a1d0134d0232313b0eb8 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java d6ca430662d456df847ac53dbd6e5632c2c936dd src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java eae17707cc420793884519b579b79652b886a696 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 2014b73a44cfdb198f738cb110399d62c396fa1f src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 010e75f05449f618017cb40fd3298e7147a1d2c8 Diff: https://reviews.apache.org/r/33611/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33537: Document Aurora security features.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/#review81864 --- Ship it! Master (c760095) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 28, 2015, 7:01 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33537/ --- (Updated April 28, 2015, 7:01 p.m.) Review request for Aurora, David McLaughlin and Bill Farner. Bugs: AURORA-817 https://issues.apache.org/jira/browse/AURORA-817 Repository: aurora Description --- SSIA A rendered version is available on github as https://github.com/kevints/aurora/blob/kts/security-docs/docs/security.md - feel free to send pull requests to fix typos, etc, from the markdown editor there and I will incorporate them into this review. Diffs - docs/deploying-aurora-scheduler.md 70d6e3d5a74cab808b327574c79f5b794aa63c28 docs/security.md PRE-CREATION Diff: https://reviews.apache.org/r/33537/diff/ Testing --- Thanks, Kevin Sweeney
Re: Review Request 33869: Fix inconsistency in MemTaskStore secondary indices.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33869/#review82614 --- Ship it! Master (751d65f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 11:16 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33869/ --- (Updated May 5, 2015, 11:16 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1305 https://issues.apache.org/jira/browse/AURORA-1305 Repository: aurora Description --- In practice this amounts to little more than a space leak, but a bug nonetheless. Diffs - src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 9c76fa58b52bacd182e87cc4ebebad6c19356a55 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java be57c5ee9f0218ed9fabb54b3579ba03477c8930 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java d1f4026f2b3d1cba542f96cecc0fe490e9ec8132 Diff: https://reviews.apache.org/r/33869/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33959: Add update watch and update start --watch flag.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/#review82944 --- Ship it! Master (27b5e25) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 7, 2015, 11:58 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/ --- (Updated May 7, 2015, 11:58 p.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Bugs: AURORA-1239 https://issues.apache.org/jira/browse/AURORA-1239 Repository: aurora Description --- Add update watch and update start --watch flag. Diffs - src/main/python/apache/aurora/client/api/__init__.py 0ae1d9fef0a7b190a51e8734c2abd71ecf3c5a32 src/main/python/apache/aurora/client/cli/update.py 8296db5434276910fc4f8470c7a962a8e8e0a9c2 src/test/python/apache/aurora/client/cli/test_supdate.py 6fb1f71b7dc787c090fce7b8e7c51baee862f336 src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 501d111f87dbaff8b6f22ffc67a9720b4dad Diff: https://reviews.apache.org/r/33959/diff/ Testing --- Unit tests + end-to-end tests. Thanks, Bill Farner
Re: Review Request 33959: Add update watch and update start --watch flag.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/#review82956 --- This patch does not apply cleanly on master (9634d25), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 12:44 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/ --- (Updated May 8, 2015, 12:44 a.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Bugs: AURORA-1239 https://issues.apache.org/jira/browse/AURORA-1239 Repository: aurora Description --- Add update watch and update start --watch flag. Diffs - src/main/python/apache/aurora/client/api/__init__.py 0ae1d9fef0a7b190a51e8734c2abd71ecf3c5a32 src/main/python/apache/aurora/client/cli/update.py 8296db5434276910fc4f8470c7a962a8e8e0a9c2 src/test/python/apache/aurora/client/cli/test_supdate.py 6fb1f71b7dc787c090fce7b8e7c51baee862f336 src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 501d111f87dbaff8b6f22ffc67a9720b4dad Diff: https://reviews.apache.org/r/33959/diff/ Testing --- Unit tests + end-to-end tests. Thanks, Bill Farner
Re: Review Request 33969: Fix broken style test cases causing PEP302
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33969/#review82975 --- Ship it! Master (018043a) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 3:30 a.m., Jake Farrell wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33969/ --- (Updated May 8, 2015, 3:30 a.m.) Review request for Aurora, Kevin Sweeney, Bill Farner, and Zameer Manji. Bugs: AURORA-1313 https://issues.apache.org/jira/browse/AURORA-1313 Repository: aurora Description --- Fix broken style test cases causing PEP302 errors Diffs - src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py 903741a1420d5c00562d75abce11c247b2e68d66 src/test/python/apache/aurora/executor/bin/test_thermos_executor_entry_point.py afd59a18fb6de9347f3ac44d78e5110ce1252fee Diff: https://reviews.apache.org/r/33969/diff/ Testing --- all tests including check style pass with this patch now Thanks, Jake Farrell
Re: Review Request 33778: Adds RPM specfile and support scripting
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33778/#review82980 --- Ship it! Master (8157181) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 5:08 a.m., Steve Salevan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33778/ --- (Updated May 8, 2015, 5:08 a.m.) Review request for Aurora, Jake Farrell, Kevin Sweeney, and Bill Farner. Bugs: AURORA-1116 https://issues.apache.org/jira/browse/AURORA-1116 Repository: aurora Description --- This review board adds support for building Red Hat-flavored packages for the Aurora scheduler alongside its client and Thermos-related binaries: aurora - Aurora Scheduler aurora-client - Aurora client and admin tool aurora-thermos - Thermos executor, runner, and observer aurora-debuginfo - Debugging symbols for Python/libmesos integration If you'd like to give it a whirl, the following Make targets will spin up new RPMs: make (nightly_srpm|nightly_rpm) - builds an RPM or SRPM with timestamped versioning, suitable for producing nightly updates make (release_srpm|release_rpm) - builds without timestamped versioning Let me know what you think, and thanks! Diffs - packaging/rpm/Makefile PRE-CREATION packaging/rpm/aurora.init.sh PRE-CREATION packaging/rpm/aurora.logrotate PRE-CREATION packaging/rpm/aurora.service PRE-CREATION packaging/rpm/aurora.spec PRE-CREATION packaging/rpm/aurora.startup.sh PRE-CREATION packaging/rpm/aurora.sysconfig PRE-CREATION packaging/rpm/clusters.json PRE-CREATION packaging/rpm/thermos-observer.init.sh PRE-CREATION packaging/rpm/thermos-observer.logrotate PRE-CREATION packaging/rpm/thermos-observer.service PRE-CREATION packaging/rpm/thermos-observer.startup.sh PRE-CREATION packaging/rpm/thermos-observer.sysconfig PRE-CREATION Diff: https://reviews.apache.org/r/33778/diff/ Testing --- Successfully executed builds on EL 6/7 and F19/20, tested deployment on EL6 and F19 Thanks, Steve Salevan
Re: Review Request 33955: Invoke pip as a python module rather than directly.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33955/#review82906 --- Ship it! Master (fef7194) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 7, 2015, 8:50 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33955/ --- (Updated May 7, 2015, 8:50 p.m.) Review request for Aurora, Brian Wickman and Zameer Manji. Bugs: AURORA-1309 https://issues.apache.org/jira/browse/AURORA-1309 Repository: aurora Description --- This is a workaround for https://github.com/pypa/pip/issues/1773 Diffs - build-support/jenkins/build.sh 229fc9d78a06f5b3a284a60504e1a1e8b50e9233 build-support/pex 61fbc915dddc7c6048862548a3348811469131d7 build-support/python/checkstyle ee8c8d5ee4cf891a9a905e48d451bcd50611bdf4 build-support/python/isort 2f697bebb5375acfddf2e1a7170b92ab9be68393 build-support/python/make-pycharm-virtualenv 84dfb64386dd9cbbe1f0cd4bb71f762928a83912 build-support/python/update-pants-requirements ae210bc8ad1175e27f903c6fc66b41f6891b78d2 pants 9dd31881deb225ded87ce40b3c0985cdf2528f7e rbt c14924954121eb03a37dfa75196fba025da4ff03 Diff: https://reviews.apache.org/r/33955/diff/ Testing --- TODO(wfarner): Report back results of end-to-end tests and invoking other tools manually. Thanks, Bill Farner
Re: Review Request 33920: Set verbose=False in the client to squelch noisy ZooKeeper logging.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33920/#review82767 --- Master (fc9cb02) is red with this patch. ./build-support/jenkins/build.sh + date Wed May 6 23:10:19 UTC 2015 + ./gradlew -Pq clean build Downloading https://services.gradle.org/distributions/gradle-2.4-bin.zip .. . .. Unzipping /home/jenkins/.gradle/wrapper/dists/gradle-2.4-bin/1lebsnfoptv8qpa10w6kyy5mp/gradle-2.4-bin.zip to /home/jenkins/.gradle/wrapper/dists/gradle-2.4-bin/1lebsnfoptv8qpa10w6kyy5mp Set executable permissions for: /home/jenkins/.gradle/wrapper/dists/gradle-2.4-bin/1lebsnfoptv8qpa10w6kyy5mp/gradle-2.4/bin/gradle :buildSrc:clean UP-TO-DATE :buildSrc:compileJava UP-TO-DATE :buildSrc:compileGroovy :buildSrc:processResources UP-TO-DATE :buildSrc:classes :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:compileTestGroovy UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build FAILURE: Build failed with an exception. * What went wrong: A problem occurred configuring root project 'aurora'. Could not resolve all dependencies for configuration ':classpath'. Could not download maven-compat.jar (org.apache.maven:maven-compat:3.0-alpha-2) Failed to download SHA1 for resource 'http://plugins.gradle.org/m2/org/apache/maven/maven-compat/3.0-alpha-2/maven-compat-3.0-alpha-2.jar'. Could not GET 'http://plugins.gradle.org/m2/org/apache/maven/maven-compat/3.0-alpha-2/maven-compat-3.0-alpha-2.jar.sha1'. peer not authenticated * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 1 mins 4.417 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 6, 2015, 11:06 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33920/ --- (Updated May 6, 2015, 11:06 p.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Repository: aurora Description --- This flag has a long thread that lands in `TwitterKazooClient`. The proposed change will mean that INFO-level ZooKeeper client logging is never shown. Zameer notes that this regressed in https://reviews.apache.org/r/32905, where he decided that the verbose kwarg
Re: Review Request 33689: Updated scheduler to process status updates asynchronously in batches.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/#review82774 --- Master (fef7194) is red with this patch. ./build-support/jenkins/build.sh :api:checkPython :api:generateThriftEntitiesJava :api:classesThriftEntities :api:compileJava UP-TO-DATE :api:generateThriftResources :api:processResources UP-TO-DATE :api:classes :api:jar :compileJava Download http://maven.twttr.com/com/google/testing/test-libraries-for-java/1.1.1/test-libraries-for-java-1.1.1.pom Download http://maven.twttr.com/com/google/testing/test-libraries-for-java/1.1.1/test-libraries-for-java-1.1.1.jar Note: Writing file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2 :processResources :classes :jar :startScripts :distTar :distZip :assemble :compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh[ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java:261:32: 'userTaskLauncher' hides a field. FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':checkstyleJmh'. Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/jmh.xml * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 3 mins 19.548 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 6, 2015, 11:34 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/ --- (Updated May 6, 2015, 11:34 p.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- Now the processing of status updates is done asynchronously with batching to insulate throughput from the expensive storage resource. Updates are placed into a queue and consumed by another thread. If many updates arrive while we're storing a batch of updates, these will be processed together in batch rather than individually. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 7bb64dd913f0fe2fede95d50a061043dbb794ab4 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 45de15a57baf7a2f7d437b590935714e28777f35 src/main/java/org/apache/aurora/scheduler/SchedulerModule.java d3ac176e9402a33fd2074b0737313458120da9e2 src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java 0ce9c9d4cf75f9add260f285115b1d60786ded57 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 4d589a33a2933b0cb6caf85abfae45c5e635c3ce src/main/java/org/apache/aurora/scheduler/mesos/Driver.java c7e45a89ceaa2c310feb610091eec0b04187860e src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java 9b8ab7c1027731f9d3f6cae77b85272ea63354d4 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java da2d5df2e053e6e1b8fb08d6813dff9eac9777f8 src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 32432322753799562d671db39c0d7fa308d962ff src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 422d5a9a42310979752eb7282658316c2b772419 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java abdeee49858fc439c27911c4eb544bf8e8c931d4 Diff: https://reviews.apache.org/r/33689/diff/ Testing --- Ran the benchmark to confirm that this improves status update throughput substantially: Before: Around 100 updates per second for a 5ms storage latency. Much worse for higher latencies. After: Around 4k-5k updates per second for a 5ms storage latency, down to 3k updates per second for 100ms storage latency. Updated unit tests for the new invariants: * TaskLaunchers are responsible for acknowledging updates. * UserTaskLauncher processes updates asynchronously. Thanks, Ben Mahler
Re: Review Request 33905: Updgrade to gradle 2.4.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33905/#review82719 --- Ship it! Master (a31acbb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 6, 2015, 7:06 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33905/ --- (Updated May 6, 2015, 7:06 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1306 https://issues.apache.org/jira/browse/AURORA-1306 Repository: aurora Description --- The code changes you see are to fix new checks introduced in the update findbugs plugin. Diffs - build.gradle 4fe4a1848742470e2f77e33c260d570acf3125ef buildSrc/gradle.properties 345242611f5fa646d82f8eaf2b05d78573e1434f config/findbugs/excludeFilter.xml 328e75c8b49696a45f113f5d28d6214f2e878ec3 gradle/wrapper/gradle-wrapper.properties 49c346d0697c648e0a91e9d27eecb02543a299a2 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 3d19831ea0eb85032172b096ac87e126701aa218 src/main/java/org/apache/aurora/scheduler/http/LeaderRedirect.java 40e0c840d761e7f4e30406c7f6b2666044dd0f37 src/test/java/org/apache/aurora/scheduler/storage/backup/StorageBackupTest.java ebe551739fb6b7132ce666ad9b3c5a86e90a5363 Diff: https://reviews.apache.org/r/33905/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33689: Updated scheduler to process status updates asynchronously in batches.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/#review82780 --- Ship it! Master (fef7194) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 7, 2015, 12:27 a.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/ --- (Updated May 7, 2015, 12:27 a.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- Now the processing of status updates is done asynchronously with batching to insulate throughput from the expensive storage resource. Updates are placed into a queue and consumed by another thread. If many updates arrive while we're storing a batch of updates, these will be processed together in batch rather than individually. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 7bb64dd913f0fe2fede95d50a061043dbb794ab4 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 45de15a57baf7a2f7d437b590935714e28777f35 src/main/java/org/apache/aurora/scheduler/SchedulerModule.java d3ac176e9402a33fd2074b0737313458120da9e2 src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java 0ce9c9d4cf75f9add260f285115b1d60786ded57 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 4d589a33a2933b0cb6caf85abfae45c5e635c3ce src/main/java/org/apache/aurora/scheduler/mesos/Driver.java c7e45a89ceaa2c310feb610091eec0b04187860e src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java 9b8ab7c1027731f9d3f6cae77b85272ea63354d4 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java da2d5df2e053e6e1b8fb08d6813dff9eac9777f8 src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 32432322753799562d671db39c0d7fa308d962ff src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 422d5a9a42310979752eb7282658316c2b772419 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java abdeee49858fc439c27911c4eb544bf8e8c931d4 Diff: https://reviews.apache.org/r/33689/diff/ Testing --- Ran the benchmark to confirm that this improves status update throughput substantially: Before: Around 100 updates per second for a 5ms storage latency. Much worse for higher latencies. After: Around 4k-5k updates per second for a 5ms storage latency, down to 3k updates per second for 100ms storage latency. Updated unit tests for the new invariants: * TaskLaunchers are responsible for acknowledging updates. * UserTaskLauncher processes updates asynchronously. Thanks, Ben Mahler
Re: Review Request 33924: Adding missing API hook support in kill commands.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33924/#review82785 --- Ship it! Master (fef7194) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 7, 2015, 1:28 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33924/ --- (Updated May 7, 2015, 1:28 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1307 https://issues.apache.org/jira/browse/AURORA-1307 Repository: aurora Description --- Added missing config options and refactored relevant tests. Diffs - src/main/python/apache/aurora/client/cli/context.py bd40410ccd0b02d0358e46eef015ce5d625c239f src/main/python/apache/aurora/client/cli/jobs.py 6d82a7e7d97f7bf8621669ce76e24c9f4c1a371d src/main/python/apache/aurora/client/cli/options.py d175e8cc061c305fe95ccfdc00f4b91cfa30ad96 src/test/python/apache/aurora/client/cli/test_cancel_update.py f9facb6dfd22b578359a68435d3e72318f5f158c src/test/python/apache/aurora/client/cli/test_kill.py 09734a7981b0f41abd0c8667caf2b121a3205853 src/test/python/apache/aurora/client/cli/util.py 1fbf05009c298dfb854cec7b699e557eb1499a43 Diff: https://reviews.apache.org/r/33924/diff/ Testing --- ./pants test.pytest --no-fast src/test/python:all Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33608: Added a status update throughput benchmark.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/#review82598 --- Master (d3b6e13) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 10:24 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/ --- (Updated May 5, 2015, 10:24 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1283 https://issues.apache.org/jira/browse/AURORA-1283 Repository: aurora Description --- In order to justify doing asynchronous batch acknowledgements and to better understand status update throughput, this introduces a benchmark. Note that this assumes that status update processing is not synchronous, so that the benchmark doesn't need to be updated for AURORA-1228. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java c54619f7cd617b48069363173dcf63b6254b4095 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java d7d659bb13f085ff06291ef0033572f8bdf29874 Diff: https://reviews.apache.org/r/33608/diff/ Testing --- Ran the benchmarks against the existing code and some pending code I have for AURORA-1228 to demonstrate the improvement. The end to end tests are broken, appears to be unrelated to my change from what I can tell. Thanks, Ben Mahler
Re: Review Request 33612: Add a task store implementation that uses a relational database.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/#review82558 --- Ship it! Master (8a1c8bb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 6:21 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/ --- (Updated May 5, 2015, 6:21 p.m.) Review request for Aurora, Joshua Cohen, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add a task store implementation that uses a relational database. The meat of this review is in `DbTaskStore`, `TaskConfigManager`, and the mapper xml files. Some supporting actors include files under views/, which serve DB business objects. I suggest reviewers start by skimming `DbTaskStore` and digesting the comments in there. There are some known loose ends here, most notably being continued performance enhancements and cleanup of relations in different tables. I've included several relevant TODOs, ~all of which must be addressed before this becomes the default task store. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 0182ecb3942cfa2d9dd21138779815f4500339be examples/vagrant/upstart/aurora-scheduler.conf cc4864c4d954d58e9bfaaaf5fc5afc8d0e032e78 src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java ed1171d52655fef643330f34913c256f77300fa2 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 3d19831ea0eb85032172b096ac87e126701aa218 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java 94ce5c3499ced1b63abf19787acc21b2cd4d0c75 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java c8df88b77b9a2017c48bdc0c9a0477927ba0b179 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 1a6c3f21d4fcb476539f588facecd8ef37332837 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/ContainerShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/TaskConstraintShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ScheduleStatusTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 4d0c10d60037a3310226a6fd8936b84ae4135e7e src/main/java/org/apache/aurora/scheduler/storage/db/views/AssignedPort.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskConfigRow.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskLink.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 21f7d4db821930d2c5b52648e1996aa1ef12a85c src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f76f9a988669dab598e9d19f849018c6f55ce03e src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java f18619a27eeb2aea8dcf01e54c23ed7d1c7d3d87 src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java bad9eb56b33c3e649c3b173e83d9c30da8f9317d Diff: https://reviews.apache.org/r/33612/diff/ Testing --- Unit tests and end-to-end tests, both using the new task store by default with this change. Thanks, Bill Farner
Re: Review Request 33991: Add a StatsProvider binding to UpdateStoreBenchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33991/#review83040 --- Ship it! Master (8157181) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 6:10 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33991/ --- (Updated May 8, 2015, 6:10 p.m.) Review request for Aurora and Joshua Cohen. Bugs: AURORA-1315 https://issues.apache.org/jira/browse/AURORA-1315 Repository: aurora Description --- Add a StatsProvider binding to UpdateStoreBenchmarks. Diffs - src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java 4cc53d8bc9a64a8f5d574e53f9ad98f37cdeff21 Diff: https://reviews.apache.org/r/33991/diff/ Testing --- Successfully ran `./gradlew jmh -Pbenchmarks='UpdateStoreBenchmarks'`. Thanks, Bill Farner
Re: Review Request 33999: Add missing binding to ThriftApiBenchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33999/#review83088 --- Ship it! Master (288f00b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 8:45 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33999/ --- (Updated May 8, 2015, 8:45 p.m.) Review request for Aurora and Joshua Cohen. Repository: aurora Description --- Sorry for the whack-a-mole, these tests take a long time to run so i was selecting specific ones to run. I'll make sure the full suite is green before i push this one. Diffs - src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java ed1171d52655fef643330f34913c256f77300fa2 Diff: https://reviews.apache.org/r/33999/diff/ Testing --- TODO(wfarner): Report back on result of ./gradlew jmh. Thanks, Bill Farner
Re: Review Request 33959: Add update wait and update start --wait flag.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/#review83104 --- Ship it! Master (288f00b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 8, 2015, 9:07 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33959/ --- (Updated May 8, 2015, 9:07 p.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Bugs: AURORA-1239 https://issues.apache.org/jira/browse/AURORA-1239 Repository: aurora Description --- Add update watch and update start --watch flag. Diffs - src/main/python/apache/aurora/client/api/__init__.py 0ae1d9fef0a7b190a51e8734c2abd71ecf3c5a32 src/main/python/apache/aurora/client/cli/update.py 7bd1eb56e07f8d67e9bf62a0d46713cbaa91aa76 src/test/python/apache/aurora/client/cli/test_supdate.py 968d456927de8044b8a8c5ca365464a8fedd3711 src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 501d111f87dbaff8b6f22ffc67a9720b4dad Diff: https://reviews.apache.org/r/33959/diff/ Testing --- Unit tests + end-to-end tests. Thanks, Bill Farner
Re: Review Request 34046: Upgade h2 to 1.4.187.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34046/#review83253 --- Master (339840d) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 11, 2015, 5:58 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34046/ --- (Updated May 11, 2015, 5:58 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1311 https://issues.apache.org/jira/browse/AURORA-1311 Repository: aurora Description --- Upgade h2 to 1.4.187. Diffs - build.gradle 829b81392d6eaf9f1eb7ae1ee89ff60b4fb32731 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java c8df88b77b9a2017c48bdc0c9a0477927ba0b179 Diff: https://reviews.apache.org/r/34046/diff/ Testing --- I ran the benchmarks before and after this change. In nearly every case, performance is marginally better (though the difference is generally within 1 stddev). Note, however, that i needed to bump the heap size on the benchmark suite. This is because my first run encountered an OOM on one microbenchmark after the change. I have no evidence whether this is because the new H2 is more memory-hungry, or if that happens occasionally and it was poor luck. Benchmark results before (old H2 version): ``` SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark 551189.355 ± 14341.947 ops/s SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark 51502.506 ± 975.321 ops/s SchedulingBenchmarks.LimitConstraintMismatchSchedulingBenchmark.runBenchmark 4427.202 ± 162.303 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 52.291 ± 1.402 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 50.159 ± 1.907 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 52.891 ± 0.643 ops/s SchedulingBenchmarks.PreemptorSlotSearchBenchmark.runBenchmark 50.845 ± 0.689 ops/s SchedulingBenchmarks.ValueConstraintMismatchSchedulingBenchmark.runBenchmark 51955.901 ± 1420.778 ops/s StatusUpdateBenchmark.runBenchmark 0.176 ± 0.019 ops/s StatusUpdateBenchmark.runBenchmark 0.038 ± 0.001 ops/s StatusUpdateBenchmark.runBenchmark 0.010 ± 0.001 ops/s ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2025.121 ± 30.865 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 115.166 ± 10.361 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 9.490 ± 4.451 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 0.241 ± 0.277 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2010.467 ± 161.806 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2009.343 ± 172.806 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2006.227 ± 86.458 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2008.623 ± 14.901 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 4368.957 ± 53.245 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 3932.150 ± 319.144 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 2069.126 ± 33.426 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 224.580 ± 10.336 ThriftApiBenchmarks.GetAllTasksBenchmark.run 5 19.286 ± 0.713 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 6287.064 ± 156.134 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 533.524 ± 101.918 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 12.988 ± 0.856 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 2.309 ± 0.105 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 5581.662 ± 366.345 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 5487.590 ± 179.228 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 5337.478 ± 113.176 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 5499.892 ± 166.960 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 13288.963 ± 661.267 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 11349.901 ± 422.637 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 5984.420 ± 454.136 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 995.175 ± 84.046 ThriftApiBenchmarks.GetRoleSummaryBenchmark.run 5 23.697 ± 1.261 UpdateStoreBenchmarks.JobDetailsBenchmark.run 325.116 ± 48.465 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run 71.977 ± 3.475 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run 37.514 ± 2.345 ops/s ``` Benchmark results after (new H2 version): ``` SchedulingBenchmarks.ClusterFullUtilizationBenchmark.runBenchmark 551152.813 ± 14762.509 ops/s SchedulingBenchmarks.InsufficientResourcesSchedulingBenchmark.runBenchmark 52547.639 ± 1618.220 ops/s
Re: Review Request 33689: Updated scheduler to process status updates asynchronously in batches.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/#review83268 --- Ship it! Master (aabb692) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 11, 2015, 6:55 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33689/ --- (Updated May 11, 2015, 6:55 p.m.) Review request for Aurora, Maxim Khutornenko and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- Now the processing of status updates is done asynchronously with batching to insulate throughput from the expensive storage resource. Updates are placed into a queue and consumed by another thread. If many updates arrive while we're storing a batch of updates, these will be processed together in batch rather than individually. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 7bb64dd913f0fe2fede95d50a061043dbb794ab4 src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java 45de15a57baf7a2f7d437b590935714e28777f35 src/main/java/org/apache/aurora/scheduler/SchedulerModule.java d3ac176e9402a33fd2074b0737313458120da9e2 src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java 0ce9c9d4cf75f9add260f285115b1d60786ded57 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 4d589a33a2933b0cb6caf85abfae45c5e635c3ce src/main/java/org/apache/aurora/scheduler/mesos/Driver.java c7e45a89ceaa2c310feb610091eec0b04187860e src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java 9b8ab7c1027731f9d3f6cae77b85272ea63354d4 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverService.java da2d5df2e053e6e1b8fb08d6813dff9eac9777f8 src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 32432322753799562d671db39c0d7fa308d962ff src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java 422d5a9a42310979752eb7282658316c2b772419 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java abdeee49858fc439c27911c4eb544bf8e8c931d4 Diff: https://reviews.apache.org/r/33689/diff/ Testing --- Ran the benchmark to confirm that this improves status update throughput substantially: Before: Around 100 updates per second for a 5ms storage latency. Much worse for higher latencies. After: Around 4k-5k updates per second for a 5ms storage latency, down to 3k updates per second for 100ms storage latency. Updated unit tests for the new invariants: * TaskLaunchers are responsible for acknowledging updates. * UserTaskLauncher processes updates asynchronously. Thanks, Ben Mahler
Re: Review Request 34015: Improve client update json output to consistently use last_modified
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34015/#review83301 --- Ship it! Master (aabb692) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 11, 2015, 10:19 p.m., Joe Smith wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34015/ --- (Updated May 11, 2015, 10:19 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1316 https://issues.apache.org/jira/browse/AURORA-1316 Repository: aurora Description --- Improve client update json output to consistently use last_modified Diffs - src/main/python/apache/aurora/client/cli/update.py 58f0db0988911858156aa086ba9fea8ecfe99143 src/test/python/apache/aurora/client/cli/test_supdate.py 158d3305c5ce5f1de5e3f4f771f318cff9bf7914 Diff: https://reviews.apache.org/r/34015/diff/ Testing --- $ ./pants test.pytest --no-fast ./src/test/python/apache/aurora/client:all src.test.python.apache.aurora.client.api.api . SUCCESS src.test.python.apache.aurora.client.api.instance_watcher . SUCCESS src.test.python.apache.aurora.client.api.job_monitor . SUCCESS src.test.python.apache.aurora.client.api.mux . SUCCESS src.test.python.apache.aurora.client.api.quota_check . SUCCESS src.test.python.apache.aurora.client.api.restarter . SUCCESS src.test.python.apache.aurora.client.api.scheduler_client . SUCCESS src.test.python.apache.aurora.client.api.sla . SUCCESS src.test.python.apache.aurora.client.api.task_util . SUCCESS src.test.python.apache.aurora.client.api.updater . SUCCESS src.test.python.apache.aurora.client.api.updater_util . SUCCESS src.test.python.apache.aurora.client.base . SUCCESS src.test.python.apache.aurora.client.binding_helper . SUCCESS src.test.python.apache.aurora.client.cli.api . SUCCESS src.test.python.apache.aurora.client.cli.client . SUCCESS src.test.python.apache.aurora.client.cli.command_hooks . SUCCESS src.test.python.apache.aurora.client.cli.config . SUCCESS src.test.python.apache.aurora.client.cli.context . SUCCESS src.test.python.apache.aurora.client.cli.cron . SUCCESS src.test.python.apache.aurora.client.cli.inspect . SUCCESS src.test.python.apache.aurora.client.cli.job . SUCCESS src.test.python.apache.aurora.client.cli.plugins . SUCCESS src.test.python.apache.aurora.client.cli.quota . SUCCESS src.test.python.apache.aurora.client.cli.sla . SUCCESS src.test.python.apache.aurora.client.cli.supdate . SUCCESS src.test.python.apache.aurora.client.cli.task . SUCCESS src.test.python.apache.aurora.client.cli.update . SUCCESS src.test.python.apache.aurora.client.cli.version . SUCCESS src.test.python.apache.aurora.client.config . SUCCESS src.test.python.apache.aurora.client.hooks.hooked_api . SUCCESS src.test.python.apache.aurora.client.hooks.non_hooked_api . SUCCESS Thanks, Joe Smith
Re: Review Request 33612: Add a task store implementation that uses a relational database.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/#review83158 --- Ship it! Master (1c09d58) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 9, 2015, 5:53 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/ --- (Updated May 9, 2015, 5:53 p.m.) Review request for Aurora, Joshua Cohen, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add a task store implementation that uses a relational database. The meat of this review is in `DbTaskStore`, `TaskConfigManager`, and the mapper xml files. Some supporting actors include files under views/, which serve DB business objects. I suggest reviewers start by skimming `DbTaskStore` and digesting the comments in there. There are some known loose ends here, most notably being continued performance enhancements and cleanup of relations in different tables. I've included several relevant TODOs, ~all of which must be addressed before this becomes the default task store. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 0182ecb3942cfa2d9dd21138779815f4500339be examples/vagrant/upstart/aurora-scheduler.conf cc4864c4d954d58e9bfaaaf5fc5afc8d0e032e78 src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 88d27dd729fd004d1510917a031591addba51816 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 239c61625bc49e53be918c59056f071b3b6b86b9 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java ea5600725d5dd84d21ca8d37b560c6d41541d016 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java c8df88b77b9a2017c48bdc0c9a0477927ba0b179 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 1a6c3f21d4fcb476539f588facecd8ef37332837 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/ContainerShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/TaskConstraintShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ScheduleStatusTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 4d0c10d60037a3310226a6fd8936b84ae4135e7e src/main/java/org/apache/aurora/scheduler/storage/db/views/AssignedPort.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskConfigRow.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskLink.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 21f7d4db821930d2c5b52648e1996aa1ef12a85c src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f76f9a988669dab598e9d19f849018c6f55ce03e src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 6a6ff27d8d0f1e537a74c1df981b97e5d8b2f2e8 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 7d856d020da854c125c037f01357e81de93895e1 src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 8f139fc987a98ef0d7f2969720134729411b8b84 src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java bad9eb56b33c3e649c3b173e83d9c30da8f9317d Diff: https://reviews.apache.org/r/33612/diff/ Testing --- Unit tests and end-to-end tests, both using the new task store by default with this change. Thanks, Bill Farner
Re: Review Request 33612: Add a task store implementation that uses a relational database.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/#review83494 --- Ship it! Master (759ef68) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 12, 2015, 9:17 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/ --- (Updated May 12, 2015, 9:17 p.m.) Review request for Aurora, Joshua Cohen, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add a task store implementation that uses a relational database. The meat of this review is in `DbTaskStore`, `TaskConfigManager`, and the mapper xml files. Some supporting actors include files under views/, which serve DB business objects. I suggest reviewers start by skimming `DbTaskStore` and digesting the comments in there. There are some known loose ends here, most notably being continued performance enhancements and cleanup of relations in different tables. I've included several relevant TODOs, ~all of which must be addressed before this becomes the default task store. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 0182ecb3942cfa2d9dd21138779815f4500339be examples/vagrant/upstart/aurora-scheduler.conf cc4864c4d954d58e9bfaaaf5fc5afc8d0e032e78 src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 88d27dd729fd004d1510917a031591addba51816 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 239c61625bc49e53be918c59056f071b3b6b86b9 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java ea5600725d5dd84d21ca8d37b560c6d41541d016 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 8859ca47088896a1814321147c6b4c31828b3db9 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 1a6c3f21d4fcb476539f588facecd8ef37332837 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 7b4067cfe10d8aa1e9e0e8bfbcd0e83c9beb4e7a src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/ContainerShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/TaskConstraintShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ScheduleStatusTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 4d0c10d60037a3310226a6fd8936b84ae4135e7e src/main/java/org/apache/aurora/scheduler/storage/db/views/AssignedPort.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskConfigRow.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskLink.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 21f7d4db821930d2c5b52648e1996aa1ef12a85c src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f76f9a988669dab598e9d19f849018c6f55ce03e src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 6a6ff27d8d0f1e537a74c1df981b97e5d8b2f2e8 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 7d856d020da854c125c037f01357e81de93895e1 src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 8f139fc987a98ef0d7f2969720134729411b8b84 src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java bad9eb56b33c3e649c3b173e83d9c30da8f9317d Diff: https://reviews.apache.org/r/33612/diff/ Testing --- Unit tests and end-to-end tests, both using the new task store by default
Re: Review Request 34121: Disable state transition logging for unknown tasks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34121/#review83484 --- Ship it! Master (759ef68) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 12, 2015, 8:40 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34121/ --- (Updated May 12, 2015, 8:40 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Quieting down state transition logging in preparation for task state reconciliation. Diffs - src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 2a943cf51d0a41260ada6965cea5f55db4c3f846 src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java 4a7ca6267ae6c4c062f941c78a4a3496d110e8c2 src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java b30a0737bb0d60cd8b58f7be0fff5db20f808347 Diff: https://reviews.apache.org/r/34121/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Maxim Khutornenko
Re: Review Request 34124: Removing dead code from StateManagerImpl.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34124/#review83501 --- Ship it! Master (759ef68) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 12, 2015, 9:42 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34124/ --- (Updated May 12, 2015, 9:42 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Removing dead code from StateManagerImpl. Diffs - src/main/java/org/apache/aurora/scheduler/state/SideEffect.java 180852e56734f5b03cf52bd8324db36612b33ade src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 2a943cf51d0a41260ada6965cea5f55db4c3f846 src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java b30a0737bb0d60cd8b58f7be0fff5db20f808347 Diff: https://reviews.apache.org/r/34124/diff/ Testing --- Thanks, Maxim Khutornenko
Re: Review Request 34307: Upgrade pants to 0.0.32
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34307/#review84019 --- Ship it! Master (07ab0bb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 16, 2015, 1:20 a.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34307/ --- (Updated May 16, 2015, 1:20 a.m.) Review request for Aurora, Joshua Cohen and Brian Wickman. Bugs: AURORA-1267 https://issues.apache.org/jira/browse/AURORA-1267 Repository: aurora Description --- Upgrade pants to 0.0.32 Diffs - .pantsversion 1fe6958562df8f355e1ca457397cb58c1c815b3f build-support/pants_requirements.txt 0c20aaf60e6a5e069f5a28d9cbe8826f869e28f4 pants.ini 99648e46ca755c3c8e22d90c7b682107d8dee333 Diff: https://reviews.apache.org/r/34307/diff/ Testing --- ./pants test.pytest --no-fast src/test/python:all ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Kevin Sweeney
Re: Review Request 34169: Add a type witness to satisfy JDK 8U11 compiler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34169/#review83616 --- Ship it! Master (bf7f9b7) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 13, 2015, 3:34 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34169/ --- (Updated May 13, 2015, 3:34 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- This may give us a green build until we can get the JDK8 upgraded on jenkins machines. Looks like this is a javac bug fixed in 8U40: http://bugs.java.com/view_bug.do?bug_id=8043926 Diffs - src/test/java/org/apache/aurora/scheduler/SchedulerLifecycleTest.java c9153e1539e5f71258b64fff19f438fe492d9600 Diff: https://reviews.apache.org/r/34169/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 34148: Enhancing the StateManager.changeState result.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34148/#review83643 --- Ship it! Master (7eba711) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 13, 2015, 5:50 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34148/ --- (Updated May 13, 2015, 5:50 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Adding more details into task state change result to facilitate task reconciliation data collection. Diffs - src/main/java/org/apache/aurora/scheduler/async/TaskTimeout.java 90e614958dfa992921e05cff86ddcc434efdd112 src/main/java/org/apache/aurora/scheduler/state/StateChangeResult.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/state/StateManager.java 71bfefb8cff3e9ad1fa9566ba55c0e3541fb01f3 src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java d87bb3818ae950125a54ff63d2ba52bfc67e6708 src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java 4a7ca6267ae6c4c062f941c78a4a3496d110e8c2 src/main/java/org/apache/aurora/scheduler/state/TransitionResult.java 874c554f84ea9290aa0d3874241da1b23dd453a7 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 160db129578365e0dd67d3354d98497f567dd621 src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 32432322753799562d671db39c0d7fa308d962ff src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java f17c43475a09bf0bbbcc49a3b372484e7937c27f src/test/java/org/apache/aurora/scheduler/async/TaskThrottlerTest.java a637101d0f01865dc2b3f0ee00aca81d0fbf0490 src/test/java/org/apache/aurora/scheduler/async/TaskTimeoutTest.java 88fc172be6c24fefb6f708ce757487082542 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 32d18a9b8af2ac04b0f82fe866c3eed7e923584d src/test/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJobTest.java 831803f0bf8000bd88fe870b6151ceca59c620fa src/test/java/org/apache/aurora/scheduler/state/MaintenanceControllerImplTest.java 7b101bc2bb5f6f1854187aa33406049a94fbb2fd src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java 15e4d388795b2ab2723373a73b419878b6346456 src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java afbca61cb6a4b0a81346c496fa21077bda3c13de src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 1ac1a2838a374383d3190d1fc5b4782d03f1d826 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 4e7ff3b3b2e12b43df157b1af6548db306c141da Diff: https://reviews.apache.org/r/34148/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 34373: Exposing task reconciliation stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34373/#review84224 --- Ship it! Master (920263b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 18, 2015, 9:41 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34373/ --- (Updated May 18, 2015, 9:41 p.m.) Review request for Aurora, Ben Mahler and Zameer Manji. Bugs: AURORA-1047 https://issues.apache.org/jira/browse/AURORA-1047 Repository: aurora Description --- Adding counters for the new `StatusChangeResult` values. Diffs - src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java f1e5dd2d47af2570f404d198b3fbdd754174206e src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java f4631c1cec4386d813d5869c28289207acf8523f Diff: https://reviews.apache.org/r/34373/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 34373: Exposing task reconciliation stats.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34373/#review84236 --- Ship it! Master (920263b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 18, 2015, 10:42 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34373/ --- (Updated May 18, 2015, 10:42 p.m.) Review request for Aurora, Ben Mahler and Zameer Manji. Bugs: AURORA-1047 https://issues.apache.org/jira/browse/AURORA-1047 Repository: aurora Description --- Adding counters for the new `StatusChangeResult` values. Diffs - src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java f1e5dd2d47af2570f404d198b3fbdd754174206e src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java f4631c1cec4386d813d5869c28289207acf8523f Diff: https://reviews.apache.org/r/34373/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 33778: Adds RPM specfile and support scripting
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33778/#review84265 --- Ship it! Master (920263b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 19, 2015, 6 a.m., Steve Salevan wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33778/ --- (Updated May 19, 2015, 6 a.m.) Review request for Aurora, Jake Farrell, Kevin Sweeney, and Bill Farner. Bugs: AURORA-1116 https://issues.apache.org/jira/browse/AURORA-1116 Repository: aurora Description --- This review board adds support for building Red Hat-flavored packages for the Aurora scheduler alongside its client and Thermos-related binaries: aurora - Aurora Scheduler aurora-client - Aurora client and admin tool aurora-thermos - Thermos executor, runner, and observer aurora-debuginfo - Debugging symbols for Python/libmesos integration If you'd like to give it a whirl, the following Make targets will spin up new RPMs: make (nightly_srpm|nightly_rpm) - builds an RPM or SRPM with timestamped versioning, suitable for producing nightly updates make (release_srpm|release_rpm) - builds without timestamped versioning Let me know what you think, and thanks! Diffs - packaging/rpm/Makefile PRE-CREATION packaging/rpm/aurora.init.sh PRE-CREATION packaging/rpm/aurora.logrotate PRE-CREATION packaging/rpm/aurora.service PRE-CREATION packaging/rpm/aurora.spec PRE-CREATION packaging/rpm/aurora.startup.sh PRE-CREATION packaging/rpm/aurora.sysconfig PRE-CREATION packaging/rpm/clusters.json PRE-CREATION packaging/rpm/thermos-observer.init.sh PRE-CREATION packaging/rpm/thermos-observer.logrotate PRE-CREATION packaging/rpm/thermos-observer.service PRE-CREATION packaging/rpm/thermos-observer.startup.sh PRE-CREATION packaging/rpm/thermos-observer.sysconfig PRE-CREATION Diff: https://reviews.apache.org/r/33778/diff/ Testing --- Successfully executed builds on EL 6/7 and F19/20, tested deployment on EL6 and F19 Thanks, Steve Salevan
Re: Review Request 33244: Fixed the end-to-end tests.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33244/#review80277 --- This patch does not apply cleanly on master (b18dc44), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 15, 2015, 10:42 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33244/ --- (Updated April 15, 2015, 10:42 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- Per a discussion with Maxim, some configuration changes broke this recently. Diffs - examples/vagrant/upstart/aurora-scheduler-kerberos.conf 7736ff4f30acdf015c8573bd17d0cc817220ac74 Diff: https://reviews.apache.org/r/33244/diff/ Testing --- Ran the end-to-end tests. Thanks, Ben Mahler
Re: Review Request 33243: Use explicit status update acknowledgements.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/#review80276 --- Master (b18dc44) is red with this patch. ./build-support/jenkins/build.sh :compileJavaNote: Writing file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2 /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java:231: error: cannot find symbol driver.acknowledgeStatusUpdate(status); ^ symbol: method acknowledgeStatusUpdate(TaskStatus) location: variable driver of type SchedulerDriver /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java:250: error: cannot find symbol driver.acknowledgeStatusUpdate(status); ^ symbol: method acknowledgeStatusUpdate(TaskStatus) location: variable driver of type SchedulerDriver /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java:39: error: no suitable constructor found for MesosSchedulerDriver(Scheduler,FrameworkInfo,String,boolean,Credential) return new MesosSchedulerDriver(scheduler, frameworkInfo, master, false, credentials.get()); ^ constructor MesosSchedulerDriver.MesosSchedulerDriver(Scheduler,FrameworkInfo,String,Credential) is not applicable (actual and formal argument lists differ in length) constructor MesosSchedulerDriver.MesosSchedulerDriver(Scheduler,FrameworkInfo,String) is not applicable (actual and formal argument lists differ in length) /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java:41: error: no suitable constructor found for MesosSchedulerDriver(Scheduler,FrameworkInfo,String,boolean) return new MesosSchedulerDriver(scheduler, frameworkInfo, master, false); ^ constructor MesosSchedulerDriver.MesosSchedulerDriver(Scheduler,FrameworkInfo,String,Credential) is not applicable (actual argument boolean cannot be converted to Credential by method invocation conversion) constructor MesosSchedulerDriver.MesosSchedulerDriver(Scheduler,FrameworkInfo,String) is not applicable (actual and formal argument lists differ in length) 4 errors FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':compileJava'. Compilation failed; see the compiler error output for details. * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 1 mins 21.608 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 15, 2015, 10:47 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33243/ --- (Updated April 15, 2015, 10:47 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1228 https://issues.apache.org/jira/browse/AURORA-1228 Repository: aurora Description --- The scheduler now explicitly acknowledges updates. I left the structure as is, per Bill's suggestion. Diffs - src/main/java/org/apache/aurora/scheduler/mesos/DriverFactoryImpl.java db7aa74c2dfca3d520d978d7837d1f2639e6c05c src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java f40746b5712f21e551c06eeb1aa379ebdcdbc693 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java 4ef23ea6ebd2c9587a8356507fcb49a36b9de219 Diff: https://reviews.apache.org/r/33243/diff/ Testing --- Ran the unit tests and the end-to-end test. Thanks, Ben Mahler
Re: Review Request 32597: Improving async preemptor efficiency.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/#review80292 --- Ship it! Master (b18dc44) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 16, 2015, 1:39 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32597/ --- (Updated April 16, 2015, 1:39 a.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- This is addressing the preemption efficiency loss due to making preemptor async. Summary: - Implemented fair task processing by evaluating pending task groups in round-robin fashion (e.g.: {G1:3, G2:2} - {G1, G2, G1, G2, G1}) against all available slaves. - Moved relevant functionality from PreemptionSlotFinder (now PreemptionVictimFilter) into PendingTaskProcessor. The bulk of new functionality is in PendingTaskIterator and PendingTaskProcessor. The rest is refactoring to adjust to the new traversal approach. Diffs - src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 00919b7910704c5025465e1071378a978e5e60a3 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionProposal.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotFinder.java f16f964f56f0f9da523950891293083f1bd86780 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 5200811ec8fe6fedf42ae2712f29051a8c0bf4a9 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorMetrics.java dc7eb4421ff305dca32f36c83605c2864fea8b11 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 7cea881a8c3c11142bd04b3c794cd86a310b15e7 src/main/java/org/apache/aurora/scheduler/base/TaskGroupKey.java 6af3949b85297043640edccc1a490906c0fcff6c src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 7bb1e7a9f27088636d6549c089b1d079dfeaf2ee src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 8a9a3b7d9686e29632f4e267f591cdb19826e0e7 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 64283fab8c61b841007d7c0a02b083b3067bc78d src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorSlotFinderTest.java eed2de99a145dd2124b7f2b4d401214f1d8adf2e Diff: https://reviews.apache.org/r/32597/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant Thanks, Maxim Khutornenko
Re: Review Request 33273: Add a specific storage routine for bulk loading data.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33273/#review80386 --- Ship it! Master (b18dc44) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 16, 2015, 9:50 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33273/ --- (Updated April 16, 2015, 9:50 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- This behavior is primarily for DbStorage, which will benefit from a signal that a mutate transaction is a bulk data load, so that it can relax transactional guarantees in the interest of recovery speed. Diffs - src/main/java/org/apache/aurora/scheduler/storage/CallOrderEnforcingStorage.java 07d81e428342b35be019aa2046c7f1554393527f src/main/java/org/apache/aurora/scheduler/storage/Storage.java 972a3c1f094dc62ebf1d35074485441181d7fe0d src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 526df103883fd720cb6c00f8dff1bb3cf7cead38 src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java 63b5b1f87ba12edbc5ad2546189dff420409d645 src/main/java/org/apache/aurora/scheduler/storage/mem/MemStorage.java dafe1c4f7c184c41dba0360f876dd8381f4aeb59 src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java 3336f8cddb2a7a6e8c9e4bc6664708342ab97979 src/test/java/org/apache/aurora/scheduler/storage/db/DbStorageTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java cb6ba25c18878b751fcaffe15977b1ca10c74d65 Diff: https://reviews.apache.org/r/33273/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33279: Add SQL tables needed for a datbase-backed task store.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33279/#review80387 --- Master (b18dc44) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 16, 2015, 10 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33279/ --- (Updated April 16, 2015, 10 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add SQL tables needed for a datbase-backed task store. Diffs - src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql ed73ef7a25537f145910dba7a1985805b3979173 Diff: https://reviews.apache.org/r/33279/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 32889: Only perform escalation wait when http teardown signal could be dispatched
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32889/#review79817 --- Ship it! Master (eb9e408) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 11, 2015, 9:52 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32889/ --- (Updated April 11, 2015, 9:52 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: AURORA-1275 https://issues.apache.org/jira/browse/AURORA-1275 Repository: aurora Description --- Only perform escalation wait when http teardown signal could be dispatched Diffs - src/main/python/apache/aurora/executor/thermos_task_runner.py 505a1e69ebfe1bd2fa74623063cb923ecff8a021 src/test/python/apache/aurora/executor/BUILD 013f056d2e131ae8077f0fcc542a340195a90e50 src/test/python/apache/aurora/executor/test_thermos_task_runner.py 6b24bbb2ab7ca16f97961aabeed945b61e5b5908 Diff: https://reviews.apache.org/r/32889/diff/ Testing --- ./pants test.pytest --no-fast --options=-v src/test/python/apache/aurora/executor:thermos_task_runner In addition, manual verification that shutdown of health-checked services without lifecycle methods is 10 seconds faster. Thanks, Stephan Erb
Re: Review Request 32541: Adding client Kerberos support.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32541/#review79765 --- This patch does not apply cleanly on master (084a622), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 10, 2015, 10:38 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32541/ --- (Updated April 10, 2015, 10:38 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Bugs: AURORA-813 https://issues.apache.org/jira/browse/AURORA-813 Repository: aurora Description --- First take on client kerberos support. The idea is to repurpose the existing auth_module system to support both legacy and kerberos during the deprecation period. This way the 0.8.0 client will be able to talk to pre-0.8.0 scheduler and use SessionKey-based authorization. Later (in 0.9.0), the payload() will be removed along with SessionKey (AURORA-1229). That will let us get rid of SchedulerProxy (or reduce it substantially). The auth_module might stay though to support other auth plugins (e.g. requests-ntlm or requests-oauthlib). TODO: integration e2e tests once scheduler side lands. Diffs - 3rdparty/python/requirements.txt 11a307cdb476ebcc25ab5c6b555bed29241ea988 examples/vagrant/aurorabuild.sh 7dc9777c054a8f8a98059f263c49ba2e3e6d7207 examples/vagrant/clusters_kerberos.json PRE-CREATION examples/vagrant/upstart/aurora-scheduler-kerberos.conf 0a809e8e0301fdb825d0d50ab2c636522fea9163 src/main/python/apache/aurora/admin/BUILD 84305f92a34564e9e8abec9250addb85141bde67 src/main/python/apache/aurora/admin/aurora_admin.py f9e8f3d19a2d861cd20aaba88455579cd8fd22ee src/main/python/apache/aurora/client/api/__init__.py a81329f6f947bbea4001c3a521c1923410a51eab src/main/python/apache/aurora/client/api/scheduler_client.py 95e553427492407743dcac31d70f392a7c1bbc02 src/main/python/apache/aurora/client/cli/BUILD c6b4e8a09d1315cf5defee2155a6e0c697892a30 src/main/python/apache/aurora/client/cli/client.py 24516d114db1743cdf600c542a27fcf5b68053a0 src/main/python/apache/aurora/common/auth/BUILD 966484627dab90e7606f1fc638cd0e159aee3317 src/main/python/apache/aurora/common/auth/__init__.py 3119fd63d3dfa28f93f219b23030059580fed098 src/main/python/apache/aurora/common/auth/auth_module.py 5f4116ef4cfbc407e0c50dc938870fb14e2299b4 src/main/python/apache/aurora/common/auth/auth_module_manager.py 73a8e5cd51edf694b971cd2c298ff406aff8c6d7 src/main/python/apache/aurora/common/auth/kerberos.py PRE-CREATION src/main/python/apache/aurora/common/transport.py 395f8a94d9a27aad00166a17f2528a8c0833ffdd src/test/python/apache/aurora/client/api/test_scheduler_client.py 0a6194831c332a96eab62b869c4e05cfa9def058 src/test/python/apache/aurora/common/test_transport.py b78e0b3badfbbeecefff7b5954f3796cef4da9d8 src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh 47d22eeacef06c999dda984f6d1ed22e5ab0c417 Diff: https://reviews.apache.org/r/32541/diff/ Testing --- ./pants test.pytest --no-fast src/test/python:all ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Maxim Khutornenko
Re: Review Request 32541: Adding client Kerberos support.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32541/#review79770 --- Ship it! Master (084a622) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 10, 2015, 10:45 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32541/ --- (Updated April 10, 2015, 10:45 p.m.) Review request for Aurora, Kevin Sweeney and Brian Wickman. Bugs: AURORA-813 https://issues.apache.org/jira/browse/AURORA-813 Repository: aurora Description --- First take on client kerberos support. The idea is to repurpose the existing auth_module system to support both legacy and kerberos during the deprecation period. This way the 0.8.0 client will be able to talk to pre-0.8.0 scheduler and use SessionKey-based authorization. Later (in 0.9.0), the payload() will be removed along with SessionKey (AURORA-1229). That will let us get rid of SchedulerProxy (or reduce it substantially). The auth_module might stay though to support other auth plugins (e.g. requests-ntlm or requests-oauthlib). TODO: integration e2e tests once scheduler side lands. Diffs - 3rdparty/python/requirements.txt 11a307cdb476ebcc25ab5c6b555bed29241ea988 examples/vagrant/aurorabuild.sh f11488a3bd04c300e2772831c4926de75053eb3f examples/vagrant/clusters_kerberos.json PRE-CREATION examples/vagrant/upstart/aurora-scheduler-kerberos.conf 0a809e8e0301fdb825d0d50ab2c636522fea9163 src/main/python/apache/aurora/admin/BUILD c51a90b08efa39e517c1ae06fe53daa013c94971 src/main/python/apache/aurora/admin/aurora_admin.py f9e8f3d19a2d861cd20aaba88455579cd8fd22ee src/main/python/apache/aurora/client/api/__init__.py a81329f6f947bbea4001c3a521c1923410a51eab src/main/python/apache/aurora/client/api/scheduler_client.py 95e553427492407743dcac31d70f392a7c1bbc02 src/main/python/apache/aurora/client/cli/BUILD 3ab41cc26b04371e7df09983ff4c4d003988cd28 src/main/python/apache/aurora/client/cli/client.py 24516d114db1743cdf600c542a27fcf5b68053a0 src/main/python/apache/aurora/common/auth/BUILD 966484627dab90e7606f1fc638cd0e159aee3317 src/main/python/apache/aurora/common/auth/__init__.py 3119fd63d3dfa28f93f219b23030059580fed098 src/main/python/apache/aurora/common/auth/auth_module.py 5f4116ef4cfbc407e0c50dc938870fb14e2299b4 src/main/python/apache/aurora/common/auth/auth_module_manager.py 73a8e5cd51edf694b971cd2c298ff406aff8c6d7 src/main/python/apache/aurora/common/auth/kerberos.py PRE-CREATION src/main/python/apache/aurora/common/transport.py 395f8a94d9a27aad00166a17f2528a8c0833ffdd src/test/python/apache/aurora/client/api/test_scheduler_client.py 0a6194831c332a96eab62b869c4e05cfa9def058 src/test/python/apache/aurora/common/test_transport.py b78e0b3badfbbeecefff7b5954f3796cef4da9d8 src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh 47d22eeacef06c999dda984f6d1ed22e5ab0c417 Diff: https://reviews.apache.org/r/32541/diff/ Testing --- ./pants test.pytest --no-fast src/test/python:all ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Maxim Khutornenko
Re: Review Request 32990: Improving JobUpdateDetails fetch performance.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32990/#review79441 --- Master (35f3e8d) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 8, 2015, 9:45 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32990/ --- (Updated April 8, 2015, 9:45 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1263 https://issues.apache.org/jira/browse/AURORA-1263 Repository: aurora Description --- Using event collections with select avoids unnecessary row explosion and provdes subsecond execution time for a large job update that used to take half a minute. Diffs - src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml 4536cd35ffc231a5d2f6c1b5bf2aaa084ce3ad1d Diff: https://reviews.apache.org/r/32990/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33019: Added new (optional) paramter to ProcessBase for a statsd end-point, and submits a counter (increment) per process.state when statsd end-point provided
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33019/#review79521 --- Master (6384710) is red with this patch. ./build-support/jenkins/build.sh :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 5 mins 5.158 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1)) Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2)) pip can't proceed with requirement 'CherryPy==3.6.0 (from -r /dev/fd/63 (line 2))' due to a pre-existing build directory. location: /tmp/user/2395/pip_build_jenkins/CherryPy This is likely due to a previous installation that failed. pip is being responsible and not assuming it can delete this. Please delete it and try again. Storing debug log for failure in /home/jenkins/.pip/pip.log I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 9, 2015, 4:38 p.m., Michael Schenck wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33019/ --- (Updated April 9, 2015, 4:38 p.m.) Review request for Aurora. Repository: aurora Description --- * Plotting {Process.name()}.{ProcessState} counter * Added unit test for writing state transitions to statsd end-point * Added statsd (optional) attribute to Process schema Diffs - src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/main/python/apache/thermos/core/process.py 5ce138dab161d880c0bd58b87a6f5a54d4ca2f99 src/main/python/apache/thermos/core/runner.py f949f279a071c6464b026749f51afc776102f2aa src/test/python/apache/thermos/core/test_process.py 5e6ad2fca616b840299bd9ca1614c82c5c39e992 Diff: https://reviews.apache.org/r/33019/diff/ Testing --- Thanks, Michael Schenck
Re: Review Request 32990: Improving JobUpdateDetails fetch performance.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32990/#review79556 --- Master (7e3e7c9) is red with this patch. ./build-support/jenkins/build.sh make[1]: Leaving directory `/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/thrift/thrift-0.9.1' make: Leaving directory `/home/jenkins/jenkins-slave/workspace/AuroraBot/build-support/thrift' :api:classesThriftNote: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. :api:checkPython :api:generateThriftEntitiesJava :api:classesThriftEntities :api:compileJava UP-TO-DATE :api:generateThriftResources :api:processResources UP-TO-DATE :api:classes :api:jar :compileJavaNote: Writing file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2 :processResources :classes :jar :startScripts :distTar :distZip :assemble :compileJmhJava :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh[ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java:67: Wrong order for 'com.twitter.common.inject.Bindings' import. Order should be: java, javax, scala, com, net, org. Each group should be separated by a single blank line. FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':checkstyleJmh'. Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/jmh.xml * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 1 mins 39.707 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 9, 2015, 6:43 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32990/ --- (Updated April 9, 2015, 6:43 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1263 https://issues.apache.org/jira/browse/AURORA-1263 Repository: aurora Description --- Using event collections with select avoids unnecessary row explosion and provdes subsecond execution time for a large job update that used to take half a minute. Before: ``` Benchmark (instances) Mode Cnt Score Error Units UpdateStoreBenchmarks.JobDetailsBenchmark.run 1000 thrpt5 15.595 ± 1.056 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run 5000 thrpt5 3.143 ± 0.259 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run1 thrpt5 1.539 ± 0.075 ops/s ``` After: ``` Benchmark (instances) Mode Cnt ScoreError Units UpdateStoreBenchmarks.JobDetailsBenchmark.run 1000 thrpt5 278.405 ± 57.311 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run 5000 thrpt5 65.386 ± 6.455 ops/s UpdateStoreBenchmarks.JobDetailsBenchmark.run1 thrpt5 33.353 ± 1.974 ops/s ``` Diffs - src/jmh/java/org/apache/aurora/benchmark/UpdateStoreBenchmarks.java PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml 4536cd35ffc231a5d2f6c1b5bf2aaa084ce3ad1d Diff: https://reviews.apache.org/r/32990/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33026: Revert Make health check configurable
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33026/#review79542 --- Master (6384710) is red with this patch. ./build-support/jenkins/build.sh :check :build :api:assemble :api:compileTestJava UP-TO-DATE :api:processTestResources UP-TO-DATE :api:testClasses UP-TO-DATE :api:test UP-TO-DATE :api:check UP-TO-DATE :api:build :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 4 mins 2.865 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1)) Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2)) pip can't proceed with requirement 'CherryPy==3.6.0 (from -r /dev/fd/63 (line 2))' due to a pre-existing build directory. location: /tmp/user/2395/pip_build_jenkins/CherryPy This is likely due to a previous installation that failed. pip is being responsible and not assuming it can delete this. Please delete it and try again. Storing debug log for failure in /home/jenkins/.pip/pip.log I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 9, 2015, 6:03 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33026/ --- (Updated April 9, 2015, 6:03 p.m.) Review request for Aurora, Brian Brazil and Kevin Sweeney. Bugs: AURORA-1266 https://issues.apache.org/jira/browse/AURORA-1266 Repository: aurora Description --- This reverts commit 27a602d2c9efdd1cd2591c9c754f086c04ad0eb9. Diffs - docs/configuration-reference.md fb753ead94fcd1ed76352306391d356652934a4e src/main/python/apache/aurora/common/http_signaler.py 531f1fecbcbf8015175c7f1cb1e1c4d3e7d1268a src/main/python/apache/aurora/config/schema/base.py ec9f983564516afe542ab277d987d4d391f87e45 src/main/python/apache/aurora/executor/common/health_checker.py 03fdf0afef120c365c6ffad09e152780eed7e351 src/test/python/apache/aurora/common/test_http_signaler.py c6a21708eb997f213f4cca038891a84f6c5218d4 src/test/python/apache/aurora/executor/common/test_health_checker.py 27c71711d52f757ed1552db4accda671a6bdafdd Diff: https://reviews.apache.org/r/33026/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 32907: Generalizing preemption reservation pool.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/#review80076 --- Ship it! Master (3b29a4b) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 6:17 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32907/ --- (Updated April 14, 2015, 6:17 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-1219 https://issues.apache.org/jira/browse/AURORA-1219 Repository: aurora Description --- Current preemption model assumes tracking preemption slots/reservations by taskId. This reduces preemption efficiency as it only takes a specific taskId scheduling round to claim the PreemptionSlot and then subsequently make a slave reservation. This diff generalizes the preemption pool by TaskGroupKey. Preemption slots are now created and cached by TaskGroupKey and are available to _any_ task from the same TaskGroup. This change should also simplify the algorithm in https://reviews.apache.org/r/32597/. Diffs - src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java e87dda47a355654c66f6f54fb25a4d9a7f68422d src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java ebc520ebb4dddbc69b2b4a6f9174c1451d61887a src/main/java/org/apache/aurora/scheduler/async/preemptor/BiCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessor.java 67ad5d7f9909bc892301c19586561b6cdbfe79e6 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCache.java b5a42a0a0498f06a382576e2c3a839bbb7f1d2b1 src/main/java/org/apache/aurora/scheduler/async/preemptor/Preemptor.java 77617ec9a2bcd062d5b80cd2b4c58dc80514 src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorModule.java 1092c055588363794b37a965fb2f17a6e50d92d7 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java b61abf93eb6bb7dee336a675546bc604612b130e src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 9c47a768ec9e5c527e7f2c50332ef77433c8e088 src/test/java/org/apache/aurora/scheduler/async/preemptor/BiCacheTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/preemptor/PendingTaskProcessorTest.java 75fc16dd2796ef2d5eebea6575d141cf2a4fade0 src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptionSlotCacheTest.java 80bd13a192bda64759b5a93749ec47ddfeae504a src/test/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImplTest.java 97d6087ab92ae6d492f50ccf6c6091ec42b1c032 Diff: https://reviews.apache.org/r/32907/diff/ Testing --- ./gradlew -Pq build Manual testing in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 33105: Return Iterable from TaskStore.fetchTasks to allow for streaming.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/#review80123 --- Ship it! Master (1dc11fb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 14, 2015, 9:29 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33105/ --- (Updated April 14, 2015, 9:29 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Repository: aurora Description --- This allows the store implementation to stream if it wishes, which we might want to take advantage of when the store is backed by a database. One thing worth noting is that this change is a good example of why using wide types at call sites is important. There were many callers of TaskStore.fetchTasks that were already assigning as `Iterable`, and they were immune to this refactor. Diffs - src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java 1da35c05fb22c8fa0227a91df6a19f1b8f1aae92 src/main/java/org/apache/aurora/scheduler/async/KillRetry.java 3bb80ec99ed1f5c1d87edb59f3c9502ab6f7c706 src/main/java/org/apache/aurora/scheduler/async/RescheduleCalculator.java 0cf7fb40e5640d0ad07a7fc46c57548a3a385872 src/main/java/org/apache/aurora/scheduler/async/TaskHistoryPruner.java 985a3196ad809c7ccf30ba87a1f10cc3dacd5f49 src/main/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJob.java 3b5dcf819228297b3840cb01788c3085759f3c4a src/main/java/org/apache/aurora/scheduler/state/MaintenanceController.java 09be4dc8d6318ccfdf10397585ef7989221afd7f src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java b6a7b4aca841025c790f4b282f0490ff5bb23b2f src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java a8e3b14b3f8147f60871f9dfa84616425636e10a src/main/java/org/apache/aurora/scheduler/storage/Storage.java 6180a36255253542197234ec610eea01366135bc src/main/java/org/apache/aurora/scheduler/storage/TaskStore.java b76c937281971a505db4b352b80cbb7ccfcafcca src/main/java/org/apache/aurora/scheduler/storage/backup/Recovery.java 38764e5e7484e2fe152460ac7920be9a5799b85d src/main/java/org/apache/aurora/scheduler/storage/backup/TemporaryStorage.java 2102adb3844fc1501ff9d672a2db78a69513231b src/main/java/org/apache/aurora/scheduler/thrift/ReadOnlySchedulerImpl.java 7aef1ca786fea521256f69f6a9e7a978649a7195 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java b7d3874e9f597d93c9f623773a3a0289c2b76d6d src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 34cbd198de8e3689b77232f9ca78a2bbb6023106 Diff: https://reviews.apache.org/r/33105/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 32861: Increase executor resource epsilon to match the minimum required by Mesos
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32861/#review78889 --- Master (27a602d) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On April 4, 2015, 9:33 a.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32861/ --- (Updated April 4, 2015, 9:33 a.m.) Review request for Aurora. Bugs: AURORA-1246 https://issues.apache.org/jira/browse/AURORA-1246 Repository: aurora Description --- Increase executor resource epsilon to match the minimum required by Mesos. Mesos issues warnings for executors with 32MB RAM. By increasing the epsilon in Aurora, we shift some resources from the task to the executor. However, the total resources per container remains constant. Diffs - src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java f55d1f4fff4724b0fb651b6f0dc10d2e8f62beff src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 8ae68653427c0bdbf33751d5ed4b74afcaf98a7b Diff: https://reviews.apache.org/r/32861/diff/ Testing --- ./gradlew -Pq build Thanks, Stephan Erb
Re: Review Request 34566: Adding H2 management console.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34566/#review86718 --- Ship it! Master (ed4415c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 4, 2015, 8:56 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34566/ --- (Updated June 4, 2015, 8:56 p.m.) Review request for Aurora, Joshua Cohen and Kevin Sweeney. Bugs: AURORA-1287 https://issues.apache.org/jira/browse/AURORA-1287 Repository: aurora Description --- Adding support for connecting to H2 DB via management console: http://www.h2database.com/html/quickstart.html The H2 console servlet is put behind Shiro. Users are expected to have `h2_management_console` permission or be a member of admin role to access the console. Vagrant example JDBC URL: jdbc:h2:mem:aurora Diffs - src/main/java/org/apache/aurora/scheduler/http/H2ConsoleModule.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 16515f6ce4518e0d7eec27dfcde6fe036ee05d1d src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java 079ff5d2c30cc94bf4aef97cd4d791d480ebeab9 src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroIniParser.java 0163ba1a6df647a644948a42f772bd838c2146ef src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilter.java 28e6b98b39a073251bfae483575edd9635ae783a src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosPermissiveAuthenticationFilter.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/H2ConsoleModuleIT.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityIT.java 6743d06be828247b1c3a232aea145b0a0c492eca src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptorTest.java 33783c88b7c6c1a7343f3271523a7db39c73e4a1 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilterTest.java e335a439a6ccc301cf7c93e138c90660fbfdc6d0 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosPermissiveAuthenticationFilterTest.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh 6e9e3b2a8d731e51d0d55d0a5075b6d2d51a02ac Diff: https://reviews.apache.org/r/34566/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh Connecting to H2 in Vagrant. Thanks, Maxim Khutornenko
Re: Review Request 35219: Relax requirement for .ini file sections.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35219/#review87069 --- Ship it! Master (65739bb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 8, 2015, 7:10 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35219/ --- (Updated June 8, 2015, 7:10 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1342 https://issues.apache.org/jira/browse/AURORA-1342 Repository: aurora Description --- Relax requirement for .ini file sections. Diffs - src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroIniParser.java 671b14fe4bcab933c2aaf3dff43ed6f9ef9c05fc src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroIniParserTest.java cb8b1add6b47a117b1f524759af2a35477a47561 src/test/resources/org/apache/aurora/scheduler/http/api/security/shiro-malformed-missing-sections.ini 5303c6e5d969c85025b36319f6e3d84a7da31ce3 src/test/resources/org/apache/aurora/scheduler/http/api/security/shiro-malformed-no-sections.ini PRE-CREATION Diff: https://reviews.apache.org/r/35219/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 34566: Adding H2 management console.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34566/#review86706 --- Ship it! Master (ed4415c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 4, 2015, 7:31 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34566/ --- (Updated June 4, 2015, 7:31 p.m.) Review request for Aurora, Joshua Cohen and Kevin Sweeney. Bugs: AURORA-1287 https://issues.apache.org/jira/browse/AURORA-1287 Repository: aurora Description --- Adding support for connecting to H2 DB via management console: http://www.h2database.com/html/quickstart.html The H2 console servlet is put behind Shiro. Users are expected to have `h2_management_console` permission or be a member of admin role to access the console. Vagrant example JDBC URL: jdbc:h2:mem:aurora Diffs - src/main/java/org/apache/aurora/scheduler/http/H2ConsoleModule.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/JettyServerModule.java 16515f6ce4518e0d7eec27dfcde6fe036ee05d1d src/main/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityModule.java 079ff5d2c30cc94bf4aef97cd4d791d480ebeab9 src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroIniParser.java 0163ba1a6df647a644948a42f772bd838c2146ef src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilter.java 28e6b98b39a073251bfae483575edd9635ae783a src/main/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosPermissiveAuthenticationFilter.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/H2ConsoleModuleIT.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/http/api/security/ApiSecurityIT.java 6743d06be828247b1c3a232aea145b0a0c492eca src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroAuthorizingParamInterceptorTest.java 33783c88b7c6c1a7343f3271523a7db39c73e4a1 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosAuthenticationFilterTest.java e335a439a6ccc301cf7c93e138c90660fbfdc6d0 src/test/java/org/apache/aurora/scheduler/http/api/security/ShiroKerberosPermissiveAuthenticationFilterTest.java PRE-CREATION src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh 6e9e3b2a8d731e51d0d55d0a5075b6d2d51a02ac Diff: https://reviews.apache.org/r/34566/diff/ Testing --- ./gradlew -Pq build ./src/test/sh/org/apache/aurora/e2e/test_kerberos_end_to_end.sh Connecting to H2 in Vagrant. Thanks, Maxim Khutornenko
Re: Review Request 34570: Use set-literals and set-comprehensions where possible
/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/cli/commands/inspect.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/cli/commands/help.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/cli/bin/thermos.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/testing/runner.py Everything Looks Good! SUCCESS: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/python/apache/thermos/testing/__init__.py Everything Looks Good! I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 21, 2015, 11:28 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34570/ --- (Updated May 21, 2015, 11:28 p.m.) Review request for Aurora, Joshua Cohen and Brian Wickman. Repository: aurora Description --- Use Python 2.7 set-literal and set-comprehension syntax where possible. There are theoretical performance gains here, for example we now often avoid the creation of an intermediate list. The main reason I did this was consistency though - as you'll see in this review there's no single agreed-upon way to do this. Diffs - 3rdparty/python/requirements.txt 439554d0980a4f3cd5b65a809547c7ac32d3cbf5 src/main/python/apache/aurora/admin/admin.py d2f7bd13ef335bdca7dc02177c031ac36574658d src/main/python/apache/aurora/admin/host_maintenance.py c31e2a6fb0348c5e7074e46d61c2d2723afaf14d src/main/python/apache/aurora/client/api/__init__.py 4b9c48e84bc203fc7b28d7efd0b2e6b8a6f18302 src/main/python/apache/aurora/client/api/instance_watcher.py 6ed815423f48301ecb3faaadcdfa1e46f50b6ca2 src/main/python/apache/aurora/client/api/job_monitor.py aa7e9760d0f4f74a5ddf97e0642bf53efc555644 src/main/python/apache/aurora/client/api/sla.py 5855685bda7188c55432dc6f1ad3c676a38309da src/main/python/apache/aurora/client/api/updater.py acbce21e991981a2e85c1a00a68e57d88c5509bb src/main/python/apache/aurora/client/cli/command_hooks.py e95b1b11e16b8e2cfc83ace8654e146be208d14e src/main/python/apache/aurora/client/cli/context.py c850223588df9d9be4a644aa51adc6419a751af0 src/main/python/apache/aurora/client/cli/jobs.py 5f608419e1235f14836865b46be05d13963deb37 src/main/python/apache/aurora/client/cli/task.py b541d03d6e34dce7f4f1e38e2d970c71c8d94e08 src/main/python/apache/aurora/config/__init__.py dd2f89014a3da730364b14e01c499ac0f2c288c1 src/main/python/apache/aurora/executor/gc_executor.py d4392faf50f8c72f08f951962913248045d7fcb5 src/main/python/apache/thermos/cli/commands/tail.py 47b52eee7cb43a46f75d62b648d293a9c603efc3 src/main/python/apache/thermos/common/planner.py da5120f8f0c2489927a03e9d78ccb4f9b6eb275f src/main/python/apache/thermos/core/helper.py 8cd32948663a5d5a1e975e1661b78de701710436 src/test/python/apache/aurora/BUILD 0701440ae47034c580b5baae40e5f6feec4bb4fd src/test/python/apache/aurora/client/api/BUILD 65b378b0b4c5fa11f9899ef04a4a10a211f37245 src/test/python/apache/aurora/client/api/test_job_monitor.py ccc8b551c15df2d726291675a90a62c30ad5ebd4 src/test/python/apache/aurora/client/api/test_sla.py 22b9f995eaa1deae9f5c5467e0939670c099f438 src/test/python/apache/aurora/client/cli/BUILD 0d85f5fba9d19ae0e9c36546f130b93664b4f6de src/test/python/apache/aurora/client/cli/test_command_hooks.py 2130f1fa71be02a004cdf8e476a270c81a7105d3 src/test/python/apache/aurora/client/cli/test_create.py 9f32ca3474ea51529217db29b88911010fa73602 src/test/python/apache/aurora/client/cli/test_plugins.py a545fece5e2b3e0017a61e1be9ac478372b1f34d src/test/python/apache/aurora/thrift_util.py PRE-CREATION Diff: https://reviews.apache.org/r/34570/diff/ Testing --- I've left most of the tests out of the scope of this review to provide more confidence; however, I've fixed them in a few places where using sets consistently broke things. Specifically I changed construction of some Thrift structs to use set-literals instead of list-literals (since their IDL type is set and not list) and this broke some tests due to a long-standing bug in Thrift. To fix them I've introduced a test dependency on [PyHamcrest](http://pyhamcrest.readthedocs.org/en/release-1.8/tutorial/) for matchers support (it's pretty much the same as Java for those who are familiar). Usage of PyHamcrest is [suggested in the mock documentation for this use case](https://docs.python.org/3/library/unittest.mock-examples.html#more-complex-argument-matching). ./pants test.pytest --no-fast src/test/python:: Thanks, Kevin Sweeney
Re: Review Request 30248: Fix a markdown syntax error in the deployment docs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30248/#review86945 --- Ship it! Master (a1f7b3d) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 7, 2015, 1:04 a.m., Jeffrey Schroeder wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30248/ --- (Updated June 7, 2015, 1:04 a.m.) Review request for Aurora, Dave Lester and Jake Farrell. Repository: aurora Description --- Fix a markdown syntax error in the deployment docs Diffs - docs/deploying-aurora-scheduler.md 7c09b3d Diff: https://reviews.apache.org/r/30248/diff/ Testing --- Rendered document before and then after. Thanks, Jeffrey Schroeder
Re: Review Request 34967: Use TaskStatus Reason to set memory limit message.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34967/#review86337 --- Master (54d561e) is red with this patch. ./build-support/jenkins/build.sh :processResources :classes :jar :startScripts :distTar :distZip :assemble :compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh :jsHint :checkstyleMain :compileTestJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java uses unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. :processTestResources :testClasses :checkstyleTest[ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:203: 'block' child have incorrect indentation level 6, expected level should be 8. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:203: 'latch' have incorrect indentation level 6, expected level should be 8. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:203: 'method call' child have incorrect indentation level 6, expected level should be 8. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:204: 'block' child have incorrect indentation level 6, expected level should be 8. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:204: 'return' have incorrect indentation level 6, expected level should be 8. [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java:205: 'block rcurly' have incorrect indentation level 4, expected level should be 6. FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':checkstyleTest'. Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/test.xml * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 1 mins 53.047 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 3, 2015, 12:58 a.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34967/ --- (Updated June 3, 2015, 12:58 a.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1341 https://issues.apache.org/jira/browse/AURORA-1341 Repository: aurora Description --- Use TaskStatus Reason to set memory limit message instead of checking the contents of the message field. Future versions of Mesos can change the diagnostic information in the message field causing Aurora to display no information when a task fails. Diffs - src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java 5af691d0ac959cf4b5d01752daf996803e91ed16 src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java fe2fc954350b42487151fc820ebad22a41aeb039 Diff: https://reviews.apache.org/r/34967/diff/ Testing --- ./gradlew test -Pq Thanks, Zameer Manji
Re: Review Request 35761: Refactoring TaskLauncher.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35761/#review89026 --- Ship it! Master (68c4620) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 23, 2015, 6:35 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35761/ --- (Updated June 23, 2015, 6:35 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1334 https://issues.apache.org/jira/browse/AURORA-1334 Repository: aurora Description --- Replacing TaskLauncher abstraction with TaskStatusUpdateManager. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java 4c63cc7af2fb5702fa649ce2cd4ee619139223e2 src/main/java/org/apache/aurora/scheduler/SchedulerModule.java ae31bdb587c0ea8380fc15c71fcf34e7e19405a6 src/main/java/org/apache/aurora/scheduler/TaskLauncher.java cd55a6ee7424873c3e615d95422c9ecab6442f46 src/main/java/org/apache/aurora/scheduler/TaskStatusHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java 6bfbf0c76399d569ac762a7b433a6c576bf0ee87 src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 17b358533853596f0a577958a3a7303fa61bf299 src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java 4f7a1bec8c32aef4b5f435c760f69de13888ea9d src/test/java/org/apache/aurora/scheduler/UserTaskLauncherTest.java 126001a90c819f9b3bfd448a060102bcef8b2a35 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java e4e158721e0bdc7ed8a02eea24f4915ab937b9c7 Diff: https://reviews.apache.org/r/35761/diff/ Testing --- Thanks, Maxim Khutornenko
Re: Review Request 35760: Removing GcExecutorLauncher code.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35760/#review89016 --- Ship it! Master (2df2db9) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 23, 2015, 6 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35760/ --- (Updated June 23, 2015, 6 p.m.) Review request for Aurora and Bill Farner. Bugs: AURORA-1334 https://issues.apache.org/jira/browse/AURORA-1334 Repository: aurora Description --- Jave side of changes for removing gc executor support. TaskLauncher refactoring will be addressed in a separate diff. Diffs - config/legacy_untested_classes.txt d2f3ca50d32abf945f58b5fb67cd27c0b127822d debian/aurora-scheduler.default 47fea5b4e6dadab9704c6ea5141268a91803fdc2 debian/aurora-scheduler.init 2ed0159d36d4be08f2c355dc1c21f36192a819f4 debian/aurora-scheduler.upstart ed7fe0917147d9c36a0828cc701da902142f72ba docs/storage-config.md 971bc1673f8be800f0d081a44018fc17a5ed025a examples/scheduler/scheduler-local.sh 6253d505f9851aea613bc95c15c313b03b57af11 examples/vagrant/upstart/aurora-scheduler-kerberos.conf 414539b1917b5d33c577f1539575934c7f7c8167 examples/vagrant/upstart/aurora-scheduler.conf f4b867cbbcdbcc792518c2f90807834e47dce253 src/main/java/org/apache/aurora/scheduler/SchedulerModule.java 6edec22aba135136c4ce4066b9535f23de077db7 src/main/java/org/apache/aurora/scheduler/async/AsyncModule.java 8bcac6c4f93eaca7250d43ab81a72102b91bf836 src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java f2ef70ddc5a859811f0e6c2ade62e115639c1654 src/test/java/org/apache/aurora/ProtobufsTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java d2ec944ca12456c23eb54c9be8b1e6756f35e7f1 src/test/java/org/apache/aurora/scheduler/base/CommandUtilTest.java ec43a44d9cbd0f303e4833953cdb65664d5ed569 Diff: https://reviews.apache.org/r/35760/diff/ Testing --- Thanks, Maxim Khutornenko
Re: Review Request 35793: DbTaskStore: delete unreferenced job keys.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35793/#review89023 --- Ship it! Master (68c4620) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 23, 2015, 6:28 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35793/ --- (Updated June 23, 2015, 6:28 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1298 https://issues.apache.org/jira/browse/AURORA-1298 Repository: aurora Description --- DbTaskStore: delete unreferenced job keys. Diffs - src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java 30e3469a9f69091b929a9243f84036fa2fdd0539 src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 3ada6286e6ef6e3302802b74eec6c46dd582dc10 src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java 7ee001f9c019a1e7b669ae5cec6088bf974a3746 src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml 8258fb102b7f5fca9635143ebaed542d43abeb9f src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 24cf52680b69e23f5ccbbcada0606975b0405d5b src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java 63a784f843eb7edf9a13c623e5355169c7e8623b src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java dda988d03634f8de582cf2b8ccdeb433c3e3de0c Diff: https://reviews.apache.org/r/35793/diff/ Testing --- Note that this removes a defensive branch wherein we checked for inbound config references before attempting to delete. With this change, we proactively delete and count on foreign key constraints to prevent deletion of rows that are still referenced. I propose we adopt this as our pattern for handling shared references, as it seems to be the most sane approach available. A gotcha with this case is that i do not believe mybatis provides a vendor-neutral approach to identify a consistency violation, and the best signal is a generic `PersistenceException`. This is unfortunate since we can't distinguish between a hopeless query with invalid syntax, a network disconnection, or the anticipated case of a consistency violation. Thanks, Bill Farner
Re: Review Request 35745: Allow equals in bound variables passed via cmdline
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35745/#review88857 --- Master (3351b09) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 22, 2015, 9:39 p.m., Michael Leinartas wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35745/ --- (Updated June 22, 2015, 9:39 p.m.) Review request for Aurora. Repository: aurora Description --- - Allow equals in bound variables passed via cmdline The parsing for the --bind parameter splits the parameter value on equals and enforces only two elements. This prevents parameters whose value contains an equals to be passed. Example case: HELLO_WORLD = Job( name = 'hello', role = 'bozo', cluster = 'west', environment = 'test', instances = 1, update_config = UpdateConfig( batch_size = 1, restart_threshold = 60, watch_secs = 45, max_per_shard_failures = 2, ), task = Task( name = 'test', processes = [Process(name = 'hello_world', cmdline = 'echo {{flags}}')], resources = Resources(cpu = 0.1, ram = 64 * MB, disk = 64 * MB), ) ) jobs = [HELLO_WORLD] aurora job create --bind flags=-myflag=value west/bozo/test/hello ./hello_world.aurora Instead, the split() should use the maxsplit parameter to limit to 1 match and leave additional equals signs in the parameter Diffs - src/main/python/apache/aurora/client/cli/options.py f8ac25287e296bcec89d3701738c7dd06ffae133 Diff: https://reviews.apache.org/r/35745/diff/ Testing --- Tried adding an equals to bind parameter (cluster_binding) in test_simple_successful_create_job_with_bindings and re-ran tests Thanks, Michael Leinartas
Re: Review Request 35745: Allow equals in bound variables passed via cmdline
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35745/#review88863 --- Master (3351b09) is red with this patch. ./build-support/jenkins/build.sh Using cached twitter.common.process-0.3.0.tar.gz Collecting twitter.common.log==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.log-0.3.0.tar.gz Collecting twitter.common.util==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.util-0.3.0.tar.gz Collecting twitter.common.collections==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.collections-0.3.0.tar.gz Collecting smmap=0.8.5 (from gitdb=0.5.1-GitPython==0.3.2.RC1-twitter.checkstyle==0.1.0) Using cached smmap-0.9.0.tar.gz Collecting twitter.common.string==0.3.0 (from twitter.common.process==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.string-0.3.0.tar.gz Collecting twitter.common.options==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.options-0.3.0.tar.gz Collecting twitter.common.dirutil==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.dirutil-0.3.0.tar.gz Collecting twitter.common.contextutil==0.3.0 (from twitter.common.util==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.contextutil-0.3.0.tar.gz Collecting twitter.common.lang==0.3.0 (from twitter.common.collections==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.lang-0.3.0.tar.gz Installing collected packages: pyflakes, pep8, smmap, gitdb, GitPython, twitter.common.lang, twitter.common.string, twitter.common.process, twitter.common.options, twitter.common.dirutil, twitter.common.log, twitter.common.contextutil, twitter.common.util, twitter.common.collections, twitter.common.app, twitter.checkstyle Running setup.py install for pyflakes Running setup.py install for pep8 Running setup.py install for smmap Running setup.py install for gitdb Running setup.py install for GitPython Running setup.py install for twitter.common.lang Running setup.py install for twitter.common.string Running setup.py install for twitter.common.process Running setup.py install for twitter.common.options Running setup.py install for twitter.common.dirutil Running setup.py install for twitter.common.log Running setup.py install for twitter.common.contextutil Running setup.py install for twitter.common.util Running setup.py install for twitter.common.collections Running setup.py install for twitter.common.app Running setup.py install for twitter.checkstyle Successfully installed GitPython-0.3.2rc1 gitdb-0.6.4 pep8-1.4.5 pyflakes-0.7.2 smmap-0.9.0 twitter.checkstyle-0.1.0 twitter.common.app-0.3.0 twitter.common.collections-0.3.0 twitter.common.contextutil-0.3.0 twitter.common.dirutil-0.3.0 twitter.common.lang-0.3.0 twitter.common.log-0.3.0 twitter.common.options-0.3.0 twitter.common.process-0.3.0 twitter.common.string-0.3.0 twitter.common.util-0.3.0 E501:ERROR src/test/python/apache/aurora/client/cli/util.py:299 line too long (114 100 characters) | processes = [Process(name = 'hello_world', cmdline = 'echo {{thermos.ports[http]}} {{flags_binding}}')], I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 22, 2015, 10:09 p.m., Michael Leinartas wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35745/ --- (Updated June 22, 2015, 10:09 p.m.) Review request for Aurora and Zameer Manji. Repository: aurora Description --- - Allow equals in bound variables passed via cmdline The parsing for the --bind parameter splits the parameter value on equals and enforces only two elements. This prevents parameters whose value contains an equals to be passed. Example case: HELLO_WORLD = Job( name = 'hello', role = 'bozo', cluster = 'west', environment = 'test', instances = 1, update_config = UpdateConfig( batch_size = 1, restart_threshold = 60, watch_secs = 45, max_per_shard_failures = 2, ), task = Task( name = 'test', processes = [Process(name = 'hello_world', cmdline = 'echo {{flags}}')], resources = Resources(cpu = 0.1, ram = 64 * MB, disk = 64 * MB), ) ) jobs = [HELLO_WORLD] aurora job create --bind
Re: Review Request 35498: Compute SLA stats for non-prod jobs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/#review88699 --- Ship it! Master (d240926) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 21, 2015, 2:39 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/ --- (Updated June 21, 2015, 2:39 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1350 https://issues.apache.org/jira/browse/AURORA-1350 Repository: aurora Description --- Compute SLA stats for non-prod jobs This is a first iteration closely following the design proposal of Maxim as posted on the mailinglist. Feedback welcome. Diffs - docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 82f36d5ca15df18bdc8ebbbd868d3394db38e603 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java ff73ca6265bd0699791da5e5b6ed4aab9156d9e4 src/main/java/org/apache/aurora/scheduler/sla/SlaModule.java 64e986fb2e0f955dd4a9c7824eac9494728bf24e src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java cb98834e925793fc116814371548a30470830164 src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 5ee123a03e3c8670e0c03b05c48a9f4c66f6af9d Diff: https://reviews.apache.org/r/35498/diff/ Testing --- `./gradlew -Pq build` and a manual verification in Vagrant. Thanks, Stephan Erb
Re: Review Request 35813: Removing GC executor code.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35813/#review89380 --- Ship it! Master (d28bd4f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 25, 2015, 4:59 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35813/ --- (Updated June 25, 2015, 4:59 p.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1333 https://issues.apache.org/jira/browse/AURORA-1333 Repository: aurora Description --- Removing GC executor code. Diffs - NEWS a17f0e7c08fd30a0b2db6814a1c755111307228b api/src/main/thrift/org/apache/aurora/gen/BUILD fe3f83b6a7680985dce01efe2d54ccc4b0c2c482 api/src/main/thrift/org/apache/aurora/gen/internal_rpc.thrift a2c230fa9b5f648c4674042411cbe46fb8bb4faa debian/aurora-executor.install 8efb1308caf64a23bed4b580de4b86e7982539e8 debian/rules 6ba18cef7fbf0989507d630a1041cdf958742617 docs/test-resource-generation.md 335586d64757f1e6293a89f14c1c3d578321eac6 examples/vagrant/aurorabuild.sh 5eb171cf45ffee1287f3ac039ab8cf3db6991a97 src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b src/main/python/apache/aurora/executor/bin/BUILD 0fbb0f1ee63499d9ce36150ae5e68fcc8a9e src/main/python/apache/aurora/executor/bin/gc_executor_main.py 8093717266f8620ebc6ef4c028ac8c87ab8d22be src/main/python/apache/aurora/executor/gc_executor.py d4392faf50f8c72f08f951962913248045d7fcb5 src/main/python/apache/thermos/cli/commands/BUILD 1dae8c981bd750807ddd1b6071e232ff2697537d src/main/python/apache/thermos/cli/commands/gc.py 23d9ff4d2048b4f2d80ea62c54e58e8d768e11c0 src/main/python/apache/thermos/cli/main.py f20f612790550b77ee3dc969c37317b014a64972 src/main/python/apache/thermos/core/BUILD efb68e84cf547cb9505a8caf5b47be394dee5145 src/main/python/apache/thermos/core/helper.py 8cd32948663a5d5a1e975e1661b78de701710436 src/main/python/apache/thermos/core/inspector.py 4fe8aa31215a12b9a53e885697b4dd4e78c1f35f src/main/python/apache/thermos/monitoring/BUILD 633dd95f9d193b1f377ab5d6cdfcdca7bdaa610f src/main/python/apache/thermos/monitoring/garbage.py aa5a2729ae6c94b6a270d97425767ccee121e588 src/test/python/apache/aurora/executor/BUILD f415ecc77022b34f053c35272d004e133803d702 src/test/python/apache/aurora/executor/bin/BUILD 2caab2aec136ede9b51ce3bdd0d139270024ba48 src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py d4c1d572663039eb742f70de1e06d708eb0b769a src/test/python/apache/aurora/executor/test_gc_executor.py 17d359054d1fc33f79a7612162064abd335ccf81 src/test/python/apache/thermos/cli/commands/test_import.py 74d9a32cf85a9e49cfbc596a7d6d44393df14e7a src/test/python/apache/thermos/monitoring/BUILD 89030d0e25e8eb3f4d4eec6a0d0a0fc3dfd43481 src/test/python/apache/thermos/monitoring/test_garbage.py 4309c46a3af5f12c8eb3192e3156348fa7c0db23 Diff: https://reviews.apache.org/r/35813/diff/ Testing --- Thanks, Maxim Khutornenko
Re: Review Request 35842: Use java.util.Optional and streams in LockManager
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35842/#review89406 --- Ship it! Master (d28bd4f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 25, 2015, 6:32 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35842/ --- (Updated June 25, 2015, 6:32 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Repository: aurora Description --- Use java.util.Optional and streams in Lock*. Taking advantage of some Java 8 features along the way, and introduced GuavaCollectors.toImmutableSet for compatibility with the streams API Diffs - src/main/java/org/apache/aurora/GuavaUtils.java 22104e6d005ae6706a06a21ffe4c730b14ecfe53 src/main/java/org/apache/aurora/scheduler/state/LockManager.java e8303f914466ebe93518ec2970e1173dc39d9adb src/main/java/org/apache/aurora/scheduler/state/LockManagerImpl.java 6aa281cb7b07e3e82c578c553eee8c7066cf2761 src/main/java/org/apache/aurora/scheduler/storage/ForwardingStore.java 2be3eb0e9405e3d34533520095423429fe00a2ca src/main/java/org/apache/aurora/scheduler/storage/LockStore.java 596a3787369616d30a2763a37bcb7b7cc39ad386 src/main/java/org/apache/aurora/scheduler/storage/db/DbLockStore.java 335d7a95e797fe940e71b10da44cbd97edea69ac src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 0670b2b4c3817becefb38f0294f111768386817b src/test/java/org/apache/aurora/GuavaUtilsTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/LockManagerImplTest.java 9c9cf1b45506a2c0e1a15e3e92f1bee83778a12a src/test/java/org/apache/aurora/scheduler/storage/db/DbLockStoreTest.java ecbe1dd1eb1f0de07bea490b076c7a920f08bccb src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 38ef412a6d29dfef7b305e00cf44522818303965 Diff: https://reviews.apache.org/r/35842/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 35886: Remove static modifier from test case.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35886/#review89414 --- Ship it! Master (ca15f25) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 25, 2015, 6:50 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35886/ --- (Updated June 25, 2015, 6:50 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Repository: aurora Description --- Remove static modifier from test case, which was introduced in the last revision to https://reviews.apache.org/r/35842/ This fixes [ant:checkstyle] /home/ksweeney/workspace/aurora/src/test/java/org/apache/aurora/GuavaUtilsTest.java:23:1: Utility classes should not have a public or default constructor. Diffs - src/test/java/org/apache/aurora/GuavaUtilsTest.java 065a4bac48b7264c1c6aaf228c684f813f2a39ea Diff: https://reviews.apache.org/r/35886/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 35901: Add a DbCronJobStore implementation.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35901/#review89466 --- Ship it! Master (56bb1e6) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 26, 2015, 2:13 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35901/ --- (Updated June 26, 2015, 2:13 a.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-415 https://issues.apache.org/jira/browse/AURORA-415 Repository: aurora Description --- This pulled a longer thread than i had hoped, but rest assured that the diff is exaggerated due to some code moves. I'll leave comments in specific places of note. Diffs - src/main/java/org/apache/aurora/scheduler/storage/db/CronJobMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/DbCronJobStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 8bfb076a80face2703045800cf5530f37d64269f src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java bb61542f4043847b1c8c92ff1b4a0ecfcfc88973 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java a5acb3ae2ca5ecfd7d470fcd02de6091d66a1694 src/main/java/org/apache/aurora/scheduler/storage/db/InsertResult.java 44dc8f5e3dcc91e80a03d980c5d8ae0db65c8b89 src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java 46fa940b9cdf0544fa33b76e87ad21045e8bcdbb src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/CronCollisionPolicyTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 0a519be65f90cb730f6bda1e6d7b019f0f15252b src/main/java/org/apache/aurora/scheduler/storage/db/views/CronJobWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java b89e7b5463cdb9ff9d1f9106dda0c0c4908225ca src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 35c83b9b8838a00693c8ebc96e496886ca249ed1 src/main/java/org/apache/aurora/scheduler/storage/mem/MemJobStore.java f9e9e89a52c9bce3dd7e5a727498a7c87c26a56c src/main/resources/org/apache/aurora/scheduler/storage/db/CronJobMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/JobKeyMapper.xml 80ff3472c768cd116770e887b68e70d2ea3c9a8d src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml e9660b732bdc97117ee467fc124e5a014f176c67 src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 0f77db7397866c008d32ea6f55c5bb577fe6468f src/test/java/org/apache/aurora/scheduler/cron/quartz/AuroraCronJobTest.java b9e16578b27de2985d24c25aae507b3540fcd3ff src/test/java/org/apache/aurora/scheduler/cron/quartz/CronIT.java 863e9c998c97506759a5526135a33a161a8fb75f src/test/java/org/apache/aurora/scheduler/cron/quartz/QuartzTestUtil.java 2d74b326831fbe22fa6045610ca8d714cd64779e src/test/java/org/apache/aurora/scheduler/storage/AbstractCronJobStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 5ad0de7c6a648f5eb6408eea7bcd789c25d55f88 src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java 31feaea7ba74350fc199333a2384419ec05f1816 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 2ed748383f269217860b80c831a3521facba83ba src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java 58256afafd12e5de234755969605861891af4daf Diff: https://reviews.apache.org/r/35901/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 35847: Split http lifecycle into a composition layer.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35847/#review89258 --- Master (d28bd4f) is red with this patch. ./build-support/jenkins/build.sh Using cached twitter.common.process-0.3.0.tar.gz Collecting twitter.common.log==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.log-0.3.0.tar.gz Collecting twitter.common.util==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.util-0.3.0.tar.gz Collecting twitter.common.collections==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.collections-0.3.0.tar.gz Collecting smmap=0.8.5 (from gitdb=0.5.1-GitPython==0.3.2.RC1-twitter.checkstyle==0.1.0) Using cached smmap-0.9.0.tar.gz Collecting twitter.common.string==0.3.0 (from twitter.common.process==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.string-0.3.0.tar.gz Collecting twitter.common.options==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.options-0.3.0.tar.gz Collecting twitter.common.dirutil==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.dirutil-0.3.0.tar.gz Collecting twitter.common.contextutil==0.3.0 (from twitter.common.util==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.contextutil-0.3.0.tar.gz Collecting twitter.common.lang==0.3.0 (from twitter.common.collections==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.lang-0.3.0.tar.gz Installing collected packages: pyflakes, pep8, smmap, gitdb, GitPython, twitter.common.lang, twitter.common.string, twitter.common.process, twitter.common.options, twitter.common.dirutil, twitter.common.log, twitter.common.contextutil, twitter.common.util, twitter.common.collections, twitter.common.app, twitter.checkstyle Running setup.py install for pyflakes Running setup.py install for pep8 Running setup.py install for smmap Running setup.py install for gitdb Running setup.py install for GitPython Running setup.py install for twitter.common.lang Running setup.py install for twitter.common.string Running setup.py install for twitter.common.process Running setup.py install for twitter.common.options Running setup.py install for twitter.common.dirutil Running setup.py install for twitter.common.log Running setup.py install for twitter.common.contextutil Running setup.py install for twitter.common.util Running setup.py install for twitter.common.collections Running setup.py install for twitter.common.app Running setup.py install for twitter.checkstyle Successfully installed GitPython-0.3.2rc1 gitdb-0.6.4 pep8-1.4.5 pyflakes-0.7.2 smmap-0.9.0 twitter.checkstyle-0.1.0 twitter.common.app-0.3.0 twitter.common.collections-0.3.0 twitter.common.contextutil-0.3.0 twitter.common.dirutil-0.3.0 twitter.common.lang-0.3.0 twitter.common.log-0.3.0 twitter.common.options-0.3.0 twitter.common.process-0.3.0 twitter.common.string-0.3.0 twitter.common.util-0.3.0 F401:ERROR src/main/python/apache/aurora/executor/thermos_task_runner.py:032 'HttpSignaler' imported but unused |from apache.aurora.common.http_signaler import HttpSignaler I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 24, 2015, 10:06 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35847/ --- (Updated June 24, 2015, 10:06 p.m.) Review request for Aurora, Brian Brazil and Maxim Khutornenko. Bugs: AURORA-1368 https://issues.apache.org/jira/browse/AURORA-1368 Repository: aurora Description --- Move shutdown endpoints to the Job config since the lifecycle is controlled by Aurora and not Thermos. Split the lifecycle management into a composition layer that can more readily be tested. Also, derp, just realized I did not update the documentation. Revision forthcoming. (Also comment on the 'union' style here -- not sure what is preferable.) Diffs - docs/configuration-reference.md 7bfd63381f54b0fe5ef6a4f17b825049b19038db src/main/python/apache/aurora/config/schema/base.py 9a6f8a16f85c324ec75352710e19249443bf2c6b src/main/python/apache/aurora/config/thrift.py 0a3e91011eccf8573feb296bd7f72913622e0ce0 src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b src/main/python/apache/aurora/executor/http_lifecycle.py PRE-CREATION src/main/python/apache/aurora
Re: Review Request 35847: Split http lifecycle into a composition layer.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35847/#review89282 --- Ship it! Master (d28bd4f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 24, 2015, 10:45 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35847/ --- (Updated June 24, 2015, 10:45 p.m.) Review request for Aurora, Brian Brazil and Maxim Khutornenko. Bugs: AURORA-1368 https://issues.apache.org/jira/browse/AURORA-1368 Repository: aurora Description --- Move shutdown endpoints to the Job config since the lifecycle is controlled by Aurora and not Thermos. Split the lifecycle management into a composition layer that can more readily be tested. Also, derp, just realized I did not update the documentation. Revision forthcoming. (Also comment on the 'union' style here -- not sure what is preferable.) Diffs - docs/configuration-reference.md 7bfd63381f54b0fe5ef6a4f17b825049b19038db src/main/python/apache/aurora/config/schema/base.py 9a6f8a16f85c324ec75352710e19249443bf2c6b src/main/python/apache/aurora/config/thrift.py 0a3e91011eccf8573feb296bd7f72913622e0ce0 src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b src/main/python/apache/aurora/executor/http_lifecycle.py PRE-CREATION src/main/python/apache/aurora/executor/thermos_task_runner.py 7bcd6c42f50665eac2e7f17b84e63f2ea7fb4d4f src/main/python/apache/thermos/config/schema_base.py a85def9eea25fa01020ca2dda4e9cefe861c4a5f src/test/python/apache/aurora/executor/BUILD f415ecc77022b34f053c35272d004e133803d702 src/test/python/apache/aurora/executor/common/fixtures.py 37d032beb66a67cfd3cfcea272747a2915a745ff src/test/python/apache/aurora/executor/test_http_lifecycle.py PRE-CREATION src/test/python/apache/aurora/executor/test_thermos_task_runner.py 3569a6abf84d5144d2e268b0a86c82285ffe2b2b Diff: https://reviews.apache.org/r/35847/diff/ Testing --- Thanks, Brian Wickman
Re: Review Request 35587: Suppress task reconciliation status update logging.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35587/#review88415 --- Ship it! Master (af7e1a7) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 18, 2015, 5:04 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35587/ --- (Updated June 18, 2015, 5:04 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Suppress task reconciliation status update logging. Diffs - src/main/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImpl.java f233d5a181bb1f43fbbfe657dbda2cf975aa6895 src/test/java/org/apache/aurora/scheduler/mesos/MesosSchedulerImplTest.java f0f9ac392973a276028aee3e06517a6e6d960bb6 Diff: https://reviews.apache.org/r/35587/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 35498: Compute SLA stats for non-prod jobs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/#review88946 --- Ship it! Master (2df2db9) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 23, 2015, 9:13 a.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/ --- (Updated June 23, 2015, 9:13 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1350 https://issues.apache.org/jira/browse/AURORA-1350 Repository: aurora Description --- Compute SLA stats for non-prod jobs This is a first iteration closely following the design proposal of Maxim as posted on the mailinglist. Feedback welcome. Diffs - docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 82f36d5ca15df18bdc8ebbbd868d3394db38e603 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java ff73ca6265bd0699791da5e5b6ed4aab9156d9e4 src/main/java/org/apache/aurora/scheduler/sla/SlaModule.java 64e986fb2e0f955dd4a9c7824eac9494728bf24e src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java cb98834e925793fc116814371548a30470830164 src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 5ee123a03e3c8670e0c03b05c48a9f4c66f6af9d Diff: https://reviews.apache.org/r/35498/diff/ Testing --- `./gradlew -Pq build` and a manual verification in Vagrant. Thanks, Stephan Erb
Re: Review Request 35498: Compute SLA stats for non-prod jobs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/#review89536 --- This patch does not apply cleanly on master (2ef6a05), do you need to rebase? I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 26, 2015, 5:45 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/ --- (Updated June 26, 2015, 5:45 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1350 https://issues.apache.org/jira/browse/AURORA-1350 Repository: aurora Description --- Compute SLA stats for non-prod jobs This is a first iteration closely following the design proposal of Maxim as posted on the mailinglist. Feedback welcome. Diffs - NEWS a17f0e7c08fd30a0b2db6814a1c755111307228b docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 82f36d5ca15df18bdc8ebbbd868d3394db38e603 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java ff73ca6265bd0699791da5e5b6ed4aab9156d9e4 src/main/java/org/apache/aurora/scheduler/sla/SlaModule.java 64e986fb2e0f955dd4a9c7824eac9494728bf24e src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java cb98834e925793fc116814371548a30470830164 src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 5ee123a03e3c8670e0c03b05c48a9f4c66f6af9d Diff: https://reviews.apache.org/r/35498/diff/ Testing --- `./gradlew -Pq build` and a manual verification in Vagrant. Thanks, Stephan Erb
Re: Review Request 35613: Fixing broken gradle dependency scanner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35613/#review88447 --- Master (e271634) is red with this patch. ./build-support/jenkins/build.sh :compileJava Download https://repo1.maven.org/maven2/org/mybatis/mybatis/3.3.0/mybatis-3.3.0.pom Download https://repo1.maven.org/maven2/org/mybatis/mybatis-parent/24/mybatis-parent-24.pom Download https://repo1.maven.org/maven2/org/slf4j/slf4j-jdk14/1.7.12/slf4j-jdk14-1.7.12.pom Download https://repo1.maven.org/maven2/org/mybatis/mybatis/3.3.0/mybatis-3.3.0.jar Download https://repo1.maven.org/maven2/org/slf4j/slf4j-jdk14/1.7.12/slf4j-jdk14-1.7.12.jar Note: Writing file:/home/jenkins/jenkins-slave/workspace/AuroraBot/dist/classes/main/com/twitter/common/args/apt/cmdline.arg.info.txt.2 :processResources :classes :jar :startScripts :distTar :distZip :assemble :compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. :processJmhResources UP-TO-DATE :jmhClasses :checkstyleJmh :jsHint :checkstyleMain[ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/auth/UnsecureAuthModule.java:40: Line is longer than 100 characters (found 118). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/auth/UnsecureAuthModule.java:41: Line is longer than 100 characters (found 119). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/auth/UnsecureAuthModule.java:42: Line is longer than 100 characters (found 119). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/auth/UnsecureAuthModule.java:43: Line is longer than 100 characters (found 110). [ant:checkstyle] /home/jenkins/jenkins-slave/workspace/AuroraBot/src/main/java/org/apache/aurora/auth/UnsecureSessionContext.java:25: 'org.apache.shiro.subject.Subject' should be separated from previous imports. FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':checkstyleMain'. Checkstyle rule violations were found. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/checkstyle/main.xml * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 2 mins 37.77 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 18, 2015, 10:12 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35613/ --- (Updated June 18, 2015, 10:12 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Our current scanner plugin version does not work with gradle 2.4. Upgrading plugin version and a few other deps (those that did not conflict with a mighty twitter.commons package): - junit: https://github.com/junit-team/junit/blob/master/doc/ReleaseNotes4.12.md - gson: https://sites.google.com/site/gson/gson-roadmap - slf4j: http://www.slf4j.org/news.html - mybatis: https://github.com/mybatis/mybatis-3/releases Diffs - build.gradle 700e1ade1470b99e9be390db150cf73aa06f17bc Diff: https://reviews.apache.org/r/35613/diff/ Testing --- ./build-support/jenkins/build.sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Maxim Khutornenko
Re: Review Request 35613: Fixing broken gradle dependency scanner.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35613/#review88565 --- Ship it! Master (589d901) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 18, 2015, 10:12 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35613/ --- (Updated June 18, 2015, 10:12 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Our current scanner plugin version does not work with gradle 2.4. Upgrading plugin version and a few other deps (those that did not conflict with a mighty twitter.commons package): - junit: https://github.com/junit-team/junit/blob/master/doc/ReleaseNotes4.12.md - gson: https://sites.google.com/site/gson/gson-roadmap - slf4j: http://www.slf4j.org/news.html - mybatis: https://github.com/mybatis/mybatis-3/releases Diffs - build.gradle 700e1ade1470b99e9be390db150cf73aa06f17bc Diff: https://reviews.apache.org/r/35613/diff/ Testing --- ./build-support/jenkins/build.sh ./src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh Thanks, Maxim Khutornenko
Re: Review Request 35839: Add exact times to update UI.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35839/#review89237 --- Master (4b8c34c) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 24, 2015, 7:05 p.m., David McLaughlin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35839/ --- (Updated June 24, 2015, 7:05 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-1371 https://issues.apache.org/jira/browse/AURORA-1371 Repository: aurora Description --- Add exact times to update UI. Diffs - src/main/resources/scheduler/assets/update.html e95053fdb499cae1c734f955d90f54cffbce68d3 Diff: https://reviews.apache.org/r/35839/diff/ Testing --- See attached screenshots. File Attachments Screen Shot 2015-06-24 at 12.03.54 PM.png https://reviews.apache.org/media/uploaded/files/2015/06/24/808ec3fd-86b2-4bfd-9f21-749efeaa6364__Screen_Shot_2015-06-24_at_12.03.54_PM.png Screen Shot 2015-06-24 at 12.04.00 PM.png https://reviews.apache.org/media/uploaded/files/2015/06/24/9ad2f767-297b-4762-9627-628a56644c5f__Screen_Shot_2015-06-24_at_12.04.00_PM.png Thanks, David McLaughlin
Re: Review Request 35813: Removing GC executor code.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35813/#review89108 --- Ship it! Master (4b8c34c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 24, 2015, 12:22 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35813/ --- (Updated June 24, 2015, 12:22 a.m.) Review request for Aurora, Bill Farner and Brian Wickman. Bugs: AURORA-1333 https://issues.apache.org/jira/browse/AURORA-1333 Repository: aurora Description --- Removing GC executor code. Diffs - api/src/main/thrift/org/apache/aurora/gen/BUILD fe3f83b6a7680985dce01efe2d54ccc4b0c2c482 api/src/main/thrift/org/apache/aurora/gen/internal_rpc.thrift a2c230fa9b5f648c4674042411cbe46fb8bb4faa debian/aurora-executor.install 8efb1308caf64a23bed4b580de4b86e7982539e8 debian/rules 6ba18cef7fbf0989507d630a1041cdf958742617 docs/test-resource-generation.md 335586d64757f1e6293a89f14c1c3d578321eac6 examples/vagrant/aurorabuild.sh 5eb171cf45ffee1287f3ac039ab8cf3db6991a97 src/main/python/apache/aurora/executor/BUILD cbb2f5f7b5daa936db71cf8c0aac8ddb2002060b src/main/python/apache/aurora/executor/bin/BUILD 0fbb0f1ee63499d9ce36150ae5e68fcc8a9e src/main/python/apache/aurora/executor/bin/gc_executor_main.py 8093717266f8620ebc6ef4c028ac8c87ab8d22be src/main/python/apache/aurora/executor/gc_executor.py d4392faf50f8c72f08f951962913248045d7fcb5 src/main/python/apache/thermos/cli/commands/BUILD 1dae8c981bd750807ddd1b6071e232ff2697537d src/main/python/apache/thermos/cli/commands/gc.py 23d9ff4d2048b4f2d80ea62c54e58e8d768e11c0 src/main/python/apache/thermos/cli/main.py f20f612790550b77ee3dc969c37317b014a64972 src/main/python/apache/thermos/core/BUILD efb68e84cf547cb9505a8caf5b47be394dee5145 src/main/python/apache/thermos/core/helper.py 8cd32948663a5d5a1e975e1661b78de701710436 src/main/python/apache/thermos/core/inspector.py 4fe8aa31215a12b9a53e885697b4dd4e78c1f35f src/main/python/apache/thermos/monitoring/BUILD 633dd95f9d193b1f377ab5d6cdfcdca7bdaa610f src/main/python/apache/thermos/monitoring/garbage.py aa5a2729ae6c94b6a270d97425767ccee121e588 src/test/python/apache/aurora/executor/BUILD f415ecc77022b34f053c35272d004e133803d702 src/test/python/apache/aurora/executor/bin/BUILD 2caab2aec136ede9b51ce3bdd0d139270024ba48 src/test/python/apache/aurora/executor/bin/test_gc_executor_entry_point.py d4c1d572663039eb742f70de1e06d708eb0b769a src/test/python/apache/aurora/executor/test_gc_executor.py 17d359054d1fc33f79a7612162064abd335ccf81 src/test/python/apache/thermos/cli/commands/test_import.py 74d9a32cf85a9e49cfbc596a7d6d44393df14e7a src/test/python/apache/thermos/monitoring/BUILD 89030d0e25e8eb3f4d4eec6a0d0a0fc3dfd43481 src/test/python/apache/thermos/monitoring/test_garbage.py 4309c46a3af5f12c8eb3192e3156348fa7c0db23 Diff: https://reviews.apache.org/r/35813/diff/ Testing --- Thanks, Maxim Khutornenko
Re: Review Request 35498: Compute SLA stats for non-prod jobs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/#review89244 --- Ship it! Master (4b8c34c) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 24, 2015, 7:09 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/ --- (Updated June 24, 2015, 7:09 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1350 https://issues.apache.org/jira/browse/AURORA-1350 Repository: aurora Description --- Compute SLA stats for non-prod jobs This is a first iteration closely following the design proposal of Maxim as posted on the mailinglist. Feedback welcome. Diffs - NEWS a17f0e7c08fd30a0b2db6814a1c755111307228b docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 82f36d5ca15df18bdc8ebbbd868d3394db38e603 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java ff73ca6265bd0699791da5e5b6ed4aab9156d9e4 src/main/java/org/apache/aurora/scheduler/sla/SlaModule.java 64e986fb2e0f955dd4a9c7824eac9494728bf24e src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java cb98834e925793fc116814371548a30470830164 src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 5ee123a03e3c8670e0c03b05c48a9f4c66f6af9d Diff: https://reviews.apache.org/r/35498/diff/ Testing --- `./gradlew -Pq build` and a manual verification in Vagrant. Thanks, Stephan Erb
Re: Review Request 35498: Compute SLA stats for non-prod jobs
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/#review89548 --- Ship it! Master (2ef6a05) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 26, 2015, 7 p.m., Stephan Erb wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35498/ --- (Updated June 26, 2015, 7 p.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1350 https://issues.apache.org/jira/browse/AURORA-1350 Repository: aurora Description --- Compute SLA stats for non-prod jobs This is a first iteration closely following the design proposal of Maxim as posted on the mailinglist. Feedback welcome. Diffs - NEWS fd5b70823cf7a695831d34453ab101c7d6a626e8 docs/sla.md 14e9108fda91200bbf56384c96b9cd926689311f src/main/java/org/apache/aurora/scheduler/sla/MetricCalculator.java 82f36d5ca15df18bdc8ebbbd868d3394db38e603 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java ff73ca6265bd0699791da5e5b6ed4aab9156d9e4 src/main/java/org/apache/aurora/scheduler/sla/SlaModule.java 64e986fb2e0f955dd4a9c7824eac9494728bf24e src/test/java/org/apache/aurora/scheduler/sla/MetricCalculatorTest.java cb98834e925793fc116814371548a30470830164 src/test/java/org/apache/aurora/scheduler/sla/SlaModuleTest.java 5ee123a03e3c8670e0c03b05c48a9f4c66f6af9d Diff: https://reviews.apache.org/r/35498/diff/ Testing --- `./gradlew -Pq build` and a manual verification in Vagrant. Thanks, Stephan Erb
Re: Review Request 32973: Allow config of the /quitquitquit and /abortabortabort endpoints.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32973/#review87546 --- Master (fab6637) is red with this patch. ./build-support/jenkins/build.sh Using cached twitter.common.process-0.3.0.tar.gz Collecting twitter.common.log==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.log-0.3.0.tar.gz Collecting twitter.common.util==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.util-0.3.0.tar.gz Collecting twitter.common.collections==0.3.0 (from twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.collections-0.3.0.tar.gz Collecting smmap=0.8.5 (from gitdb=0.5.1-GitPython==0.3.2.RC1-twitter.checkstyle==0.1.0) Using cached smmap-0.9.0.tar.gz Collecting twitter.common.string==0.3.0 (from twitter.common.process==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.string-0.3.0.tar.gz Collecting twitter.common.options==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.options-0.3.0.tar.gz Collecting twitter.common.dirutil==0.3.0 (from twitter.common.log==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.dirutil-0.3.0.tar.gz Collecting twitter.common.contextutil==0.3.0 (from twitter.common.util==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.contextutil-0.3.0.tar.gz Collecting twitter.common.lang==0.3.0 (from twitter.common.collections==0.3.0-twitter.common.app==0.3.0-twitter.checkstyle==0.1.0) Using cached twitter.common.lang-0.3.0.tar.gz Installing collected packages: pyflakes, pep8, smmap, gitdb, GitPython, twitter.common.lang, twitter.common.string, twitter.common.process, twitter.common.options, twitter.common.dirutil, twitter.common.log, twitter.common.contextutil, twitter.common.util, twitter.common.collections, twitter.common.app, twitter.checkstyle Running setup.py install for pyflakes Running setup.py install for pep8 Running setup.py install for smmap Running setup.py install for gitdb Running setup.py install for GitPython Running setup.py install for twitter.common.lang Running setup.py install for twitter.common.string Running setup.py install for twitter.common.process Running setup.py install for twitter.common.options Running setup.py install for twitter.common.dirutil Running setup.py install for twitter.common.log Running setup.py install for twitter.common.contextutil Running setup.py install for twitter.common.util Running setup.py install for twitter.common.collections Running setup.py install for twitter.common.app Running setup.py install for twitter.checkstyle Successfully installed GitPython-0.3.2rc1 gitdb-0.6.4 pep8-1.4.5 pyflakes-0.7.2 smmap-0.9.0 twitter.checkstyle-0.1.0 twitter.common.app-0.3.0 twitter.common.collections-0.3.0 twitter.common.contextutil-0.3.0 twitter.common.dirutil-0.3.0 twitter.common.lang-0.3.0 twitter.common.log-0.3.0 twitter.common.options-0.3.0 twitter.common.process-0.3.0 twitter.common.string-0.3.0 twitter.common.util-0.3.0 F401:ERROR src/test/python/apache/aurora/executor/test_thermos_task_runner.py:024 'pytest' imported but unused |import pytest I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 11, 2015, 10:09 a.m., Brian Brazil wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32973/ --- (Updated June 11, 2015, 10:09 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1257 https://issues.apache.org/jira/browse/AURORA-1257 Repository: aurora Description --- Fix name of test for sigterm being ignored. Based on https://github.com/tellapart/incubator-aurora/commit/c90532d69d69045c545004451e3660f5282e6e34 Diffs - docs/configuration-reference.md fb753ead94fcd1ed76352306391d356652934a4e src/main/python/apache/aurora/common/http_signaler.py 531f1fecbcbf8015175c7f1cb1e1c4d3e7d1268a src/main/python/apache/aurora/executor/thermos_task_runner.py 837ad5c55b4a02215a8d509c211e80529536f705 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/test/python/apache/aurora/common/test_http_signaler.py c6a21708eb997f213f4cca038891a84f6c5218d4 src/test/python/apache/aurora/executor/test_thermos_task_runner.py 1e21a112efcbd0a575fb8d90f78ff1e8c7f034cf Diff: https://reviews.apache.org/r/32973/diff/ Testing --- Unittests and manually in vagrant cluster. Also
Re: Review Request 32973: Allow config of the /quitquitquit and /abortabortabort endpoints.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32973/#review87549 --- Ship it! Master (fab6637) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On June 11, 2015, 10:39 a.m., Brian Brazil wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32973/ --- (Updated June 11, 2015, 10:39 a.m.) Review request for Aurora, Kevin Sweeney and Maxim Khutornenko. Bugs: AURORA-1257 https://issues.apache.org/jira/browse/AURORA-1257 Repository: aurora Description --- Fix name of test for sigterm being ignored. Based on https://github.com/tellapart/incubator-aurora/commit/c90532d69d69045c545004451e3660f5282e6e34 Diffs - docs/configuration-reference.md fb753ead94fcd1ed76352306391d356652934a4e src/main/python/apache/aurora/common/http_signaler.py 531f1fecbcbf8015175c7f1cb1e1c4d3e7d1268a src/main/python/apache/aurora/executor/thermos_task_runner.py 837ad5c55b4a02215a8d509c211e80529536f705 src/main/python/apache/thermos/config/schema_base.py f9143cc1b83143d6147f59d90c79435d055d0518 src/test/python/apache/aurora/common/test_http_signaler.py c6a21708eb997f213f4cca038891a84f6c5218d4 src/test/python/apache/aurora/executor/test_thermos_task_runner.py 1e21a112efcbd0a575fb8d90f78ff1e8c7f034cf Diff: https://reviews.apache.org/r/32973/diff/ Testing --- Unittests and manually in vagrant cluster. Also verified that the flaky tests pass. Thanks, Brian Brazil