Review Request 68078: Updating aurora-packaging for compatibility with Pants 1.6.0 and to reflect Mesos 1.5.0 change
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68078/ --- Review request for Aurora and Stephan Erb. Repository: aurora-packaging Description --- * Upgrading Trusty Box with python2.7 version that is has hmac.compare_digest support. * Upgradinb Virtualenv to 16.0.0 * Upgrading to Mesos 1.5.0 on all test boxes. * Upgrading Centos 7 box from 7.4 to 7.5 * Migrating from deprecated Pants --config-override to --config-files as per https://github.com/pantsbuild/pants/commit/e0d5108ff75a41421a11321c73c817332f0a1c86 Diffs - build-support/virtualenv d6484f58fbffd33ef61d6052c869c55153ec7313 builder/deb/ubuntu-trusty/Dockerfile b2b32c6be6f2b8fb61321bf700192ce4578ed2f8 specs/debian/control 83b20b67518ecfb03380839e63d20fe94d2765aa specs/debian/rules 8e338b83fa9b6c4cad07e29a4bfeac2808a13c44 specs/rpm/aurora.spec 035f049b07794722551035a99dfd014152e3396e test/deb/debian-jessie/provision.sh e381613198b1a914ccc1bf8ca724bf5947d4bdb3 test/deb/ubuntu-trusty/provision.sh 1a622889c177105fa54fac6fc8c2fb66983eef87 test/deb/ubuntu-xenial/provision.sh c55af0c1e4f3f87c93357a04ce1ca14123ce03e2 test/rpm/centos-7/Vagrantfile bea36d18e9b7aa1dac2d51edfcc789b9217550ce test/rpm/centos-7/provision.sh 103937610740332a9ce14253a0e3edc7a00b1710 Diff: https://reviews.apache.org/r/68078/diff/1/ Testing --- ./test/test-artifact.sh on all boxes and their respective releases. Thanks, Renan DelValle
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/#review206528 --- Ship it! Master (3738c3e) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing "@ReviewBot retry" - Aurora ReviewBot On July 26, 2018, 11:29 p.m., David McLaughlin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68071/ > --- > > (Updated July 26, 2018, 11:29 p.m.) > > > Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. > > > Repository: aurora > > > Description > --- > > We are running into a situation where we have a lot of short-lived ad-hoc > services launched and their updates are sticking around for 30 days, even > though the tasks are garbage collected much sooner. This change picks up > those updates and prunes them as soon as the tasks are gone. > > > Diffs > - > > > src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java > 05ada3ccba8facc63d86736199b741bfcaca9697 > > src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java > a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe > > > Diff: https://reviews.apache.org/r/68071/diff/2/ > > > Testing > --- > > ./gradlew test > > > Thanks, > > David McLaughlin > >
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/ --- (Updated July 26, 2018, 11:29 p.m.) Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. Changes --- Style fix. Repository: aurora Description --- We are running into a situation where we have a lot of short-lived ad-hoc services launched and their updates are sticking around for 30 days, even though the tasks are garbage collected much sooner. This change picks up those updates and prunes them as soon as the tasks are gone. Diffs (updated) - src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java 05ada3ccba8facc63d86736199b741bfcaca9697 src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe Diff: https://reviews.apache.org/r/68071/diff/2/ Changes: https://reviews.apache.org/r/68071/diff/1-2/ Testing --- ./gradlew test Thanks, David McLaughlin
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/#review206523 --- Master (4e28e73) is red with this patch. ./build-support/jenkins/build.sh :distZip :assemble :compileTestJavaNote: Some input files use or override a deprecated API. Note: Recompile with -Xlint:deprecation for details. Note: Some input files use unchecked or unsafe operations. Note: Recompile with -Xlint:unchecked for details. :processTestResources :testClasses :compileJmhJavaNote: /home/jenkins/jenkins-slave/workspace/AuroraBot/src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. :processJmhResources NO-SOURCE :jmhClasses :checkstyleJmh :checkstyleMain :checkstyleTest :licenseJmh UP-TO-DATE :licenseMain UP-TO-DATE :licenseTest UP-TO-DATE :license UP-TO-DATE :pmdJmh :pmdMain :pmdTest /home/jenkins/jenkins-slave/workspace/AuroraBot/src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java:81: Avoid unused local variables such as 'update8'. :pmdTest FAILED FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':pmdTest'. > 1 PMD rule violations were found. See the report at: > file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/pmd/test.html * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. * Get more help at https://help.gradle.org BUILD FAILED in 7m 53s 39 actionable tasks: 30 executed, 9 up-to-date I will refresh this build result if you post a review containing "@ReviewBot retry" - Aurora ReviewBot On July 26, 2018, 10:04 p.m., David McLaughlin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68071/ > --- > > (Updated July 26, 2018, 10:04 p.m.) > > > Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. > > > Repository: aurora > > > Description > --- > > We are running into a situation where we have a lot of short-lived ad-hoc > services launched and their updates are sticking around for 30 days, even > though the tasks are garbage collected much sooner. This change picks up > those updates and prunes them as soon as the tasks are gone. > > > Diffs > - > > > src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java > 05ada3ccba8facc63d86736199b741bfcaca9697 > > src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java > a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe > > > Diff: https://reviews.apache.org/r/68071/diff/1/ > > > Testing > --- > > ./gradlew test > > > Thanks, > > David McLaughlin > >
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/#review206521 --- Ship it! Ship It! - Santhosh Kumar Shanmugham On July 26, 2018, 3:04 p.m., David McLaughlin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68071/ > --- > > (Updated July 26, 2018, 3:04 p.m.) > > > Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. > > > Repository: aurora > > > Description > --- > > We are running into a situation where we have a lot of short-lived ad-hoc > services launched and their updates are sticking around for 30 days, even > though the tasks are garbage collected much sooner. This change picks up > those updates and prunes them as soon as the tasks are gone. > > > Diffs > - > > > src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java > 05ada3ccba8facc63d86736199b741bfcaca9697 > > src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java > a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe > > > Diff: https://reviews.apache.org/r/68071/diff/1/ > > > Testing > --- > > ./gradlew test > > > Thanks, > > David McLaughlin > >
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/ --- (Updated July 26, 2018, 10:04 p.m.) Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. Repository: aurora Description (updated) --- We are running into a situation where we have a lot of short-lived ad-hoc services launched and their updates are sticking around for 30 days, even though the tasks are garbage collected much sooner. This change picks up those updates and prunes them as soon as the tasks are gone. Diffs - src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java 05ada3ccba8facc63d86736199b741bfcaca9697 src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe Diff: https://reviews.apache.org/r/68071/diff/1/ Testing --- ./gradlew test Thanks, David McLaughlin
Re: Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/#review206520 --- Ship it! Ship It! - Jordan Ly On July 26, 2018, 10:02 p.m., David McLaughlin wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68071/ > --- > > (Updated July 26, 2018, 10:02 p.m.) > > > Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. > > > Repository: aurora > > > Description > --- > > Prune updates that have no surviving job keys in the TaskStore > > > Diffs > - > > > src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java > 05ada3ccba8facc63d86736199b741bfcaca9697 > > src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java > a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe > > > Diff: https://reviews.apache.org/r/68071/diff/1/ > > > Testing > --- > > ./gradlew test > > > Thanks, > > David McLaughlin > >
Review Request 68071: Prune updates that have no surviving job keys in the TaskStore
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68071/ --- Review request for Aurora, Jordan Ly and Santhosh Kumar Shanmugham. Repository: aurora Description --- Prune updates that have no surviving job keys in the TaskStore Diffs - src/main/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPruner.java 05ada3ccba8facc63d86736199b741bfcaca9697 src/test/java/org/apache/aurora/scheduler/pruning/JobUpdateHistoryPrunerTest.java a1bf04ab8206fc0ca301d4b1b1cbe854df209bbe Diff: https://reviews.apache.org/r/68071/diff/1/ Testing --- ./gradlew test Thanks, David McLaughlin
Re: Review Request 68047: Add size metric for memory stores, add MemSchedulerStoreTest
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68047/#review206519 --- Ship it! Ship It! - Renan DelValle On July 25, 2018, 3:37 p.m., Jordan Ly wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68047/ > --- > > (Updated July 25, 2018, 3:37 p.m.) > > > Review request for Aurora, David McLaughlin, Renan DelValle, Santhosh Kumar > Shanmugham, and Stephan Erb. > > > Repository: aurora > > > Description > --- > > Currently, we only track the size metrics for: > - # of tasks via `task_store_index_(host|job)` > - # of crons via `mem_storage_cron_size` > > I am hoping to add: > - # of attributes via `mem_storage_attributes_size` > - # of maintenance requests via `mem_storage_maintenance_size` > - # of job updates via `mem_storage_update_size` > - # of quotas via `mem_storage_quota_size` > > This will help us track the growth of stores over time. Additionally, I added > a `MemSchedulerStoreTest` since one did not exist previously and nothing was > extending the abtract version of the test. > > > Diffs > - > > > src/main/java/org/apache/aurora/scheduler/storage/mem/MemAttributeStore.java > 67684cfd9c17c6a86999a66dbd4dd9c2ef9a9938 > src/main/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStore.java > a1e1f1ef7ab3bc1b0082c31c860f144f95e78fae > > src/main/java/org/apache/aurora/scheduler/storage/mem/MemHostMaintenanceStore.java > c8d96f2cfd2fdcf8d80fd089b032dbd14a1e72b9 > > src/main/java/org/apache/aurora/scheduler/storage/mem/MemJobUpdateStore.java > 9e86b9e276ea90a249284a824705b5bbf19dcbce > src/main/java/org/apache/aurora/scheduler/storage/mem/MemQuotaStore.java > afb29fce68c432a2155d3e503d35e10c58b262be > > src/test/java/org/apache/aurora/scheduler/storage/AbstractAttributeStoreTest.java > 687fd963a5e782c7892b2cbbbcaf283653aed30f > > src/test/java/org/apache/aurora/scheduler/storage/AbstractCronJobStoreTest.java > 889cb01a0f1845ebddc44d6d737228f32665a628 > > src/test/java/org/apache/aurora/scheduler/storage/AbstractHostMaintenanceStoreTest.java > e95955c29c28d2c4474debcde2ad3fa2a9047578 > > src/test/java/org/apache/aurora/scheduler/storage/AbstractJobUpdateStoreTest.java > 6af66aa79788c07b006f163b4546c25b8ff36012 > > src/test/java/org/apache/aurora/scheduler/storage/AbstractQuotaStoreTest.java > e1d7da51503a1ea1c1748e14e439b81109b45047 > > src/test/java/org/apache/aurora/scheduler/storage/AbstractSchedulerStoreTest.java > cb8051927b66676a1e7afbbd9e1a3d10a037f429 > > src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java > c53e5847a673e398413b80fbd1a9bde9c3774cab > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemAttributeStoreTest.java > 64b19a9f2ca7c60c983b4dac1704c3056513fc2b > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java > 15e0e309f6b92ffb8b268b5e0e94e81054ee7a2c > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemHostMaintenanceStoreTest.java > ce1a9d6a0749df465129fc8fc22b9483551ca02a > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemJobUpdateStoreTest.java > cbf1bc4fe4a9ca814cd948cc99b9107c0be59615 > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemQuotaStoreTest.java > e8324eecafd91789b6ddee24300e59399641a05e > > src/test/java/org/apache/aurora/scheduler/storage/mem/MemSchedulerStoreTest.java > PRE-CREATION > > > Diff: https://reviews.apache.org/r/68047/diff/1/ > > > Testing > --- > > `./gradlew test` > > I will deploy to vagrant and ensure the new metrics are being recorded. > > > Thanks, > > Jordan Ly > >
