Review Request 33869: Fix inconsistency in MemTaskStore secondary indices.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33869/ --- Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1305 https://issues.apache.org/jira/browse/AURORA-1305 Repository: aurora Description --- In practice this amounts to little more than a space leak, but a bug nonetheless. Diffs - src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 9c76fa58b52bacd182e87cc4ebebad6c19356a55 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java be57c5ee9f0218ed9fabb54b3579ba03477c8930 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java d1f4026f2b3d1cba542f96cecc0fe490e9ec8132 Diff: https://reviews.apache.org/r/33869/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33869: Fix inconsistency in MemTaskStore secondary indices.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33869/#review82614 --- Ship it! Master (751d65f) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 11:16 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33869/ --- (Updated May 5, 2015, 11:16 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1305 https://issues.apache.org/jira/browse/AURORA-1305 Repository: aurora Description --- In practice this amounts to little more than a space leak, but a bug nonetheless. Diffs - src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 9c76fa58b52bacd182e87cc4ebebad6c19356a55 src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java be57c5ee9f0218ed9fabb54b3579ba03477c8930 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java d1f4026f2b3d1cba542f96cecc0fe490e9ec8132 Diff: https://reviews.apache.org/r/33869/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 33608: Added a status update throughput benchmark.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/#review82598 --- Master (d3b6e13) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 10:24 p.m., Ben Mahler wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33608/ --- (Updated May 5, 2015, 10:24 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1283 https://issues.apache.org/jira/browse/AURORA-1283 Repository: aurora Description --- In order to justify doing asynchronous batch acknowledgements and to better understand status update throughput, this introduces a benchmark. Note that this assumes that status update processing is not synchronous, so that the benchmark doesn't need to be updated for AURORA-1228. Diffs - src/jmh/java/org/apache/aurora/benchmark/StatusUpdateBenchmark.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeOfferManager.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeSchedulerDriver.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/UserTaskLauncher.java c54619f7cd617b48069363173dcf63b6254b4095 src/main/java/org/apache/aurora/scheduler/mesos/SchedulerDriverModule.java d7d659bb13f085ff06291ef0033572f8bdf29874 Diff: https://reviews.apache.org/r/33608/diff/ Testing --- Ran the benchmarks against the existing code and some pending code I have for AURORA-1228 to demonstrate the improvement. The end to end tests are broken, appears to be unrelated to my change from what I can tell. Thanks, Ben Mahler
Re: Review Request 33612: Add a task store implementation that uses a relational database.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/#review82558 --- Ship it! Master (8a1c8bb) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On May 5, 2015, 6:21 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/ --- (Updated May 5, 2015, 6:21 p.m.) Review request for Aurora, Joshua Cohen, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add a task store implementation that uses a relational database. The meat of this review is in `DbTaskStore`, `TaskConfigManager`, and the mapper xml files. Some supporting actors include files under views/, which serve DB business objects. I suggest reviewers start by skimming `DbTaskStore` and digesting the comments in there. There are some known loose ends here, most notably being continued performance enhancements and cleanup of relations in different tables. I've included several relevant TODOs, ~all of which must be addressed before this becomes the default task store. Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 0182ecb3942cfa2d9dd21138779815f4500339be examples/vagrant/upstart/aurora-scheduler.conf cc4864c4d954d58e9bfaaaf5fc5afc8d0e032e78 src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java ed1171d52655fef643330f34913c256f77300fa2 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 3d19831ea0eb85032172b096ac87e126701aa218 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java 94ce5c3499ced1b63abf19787acc21b2cd4d0c75 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java c8df88b77b9a2017c48bdc0c9a0477927ba0b179 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 1a6c3f21d4fcb476539f588facecd8ef37332837 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/ContainerShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/TaskConstraintShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ScheduleStatusTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 4d0c10d60037a3310226a6fd8936b84ae4135e7e src/main/java/org/apache/aurora/scheduler/storage/db/views/AssignedPort.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskConfigRow.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskLink.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 21f7d4db821930d2c5b52648e1996aa1ef12a85c src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f76f9a988669dab598e9d19f849018c6f55ce03e src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java f18619a27eeb2aea8dcf01e54c23ed7d1c7d3d87 src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java bad9eb56b33c3e649c3b173e83d9c30da8f9317d Diff: https://reviews.apache.org/r/33612/diff/ Testing --- Unit tests and end-to-end tests, both using the new task store by default with this change. Thanks, Bill Farner
Re: Review Request 33784: Symlink upstart configurations in vagrant rather than copying.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33784/#review82549 --- Ship it! Ship It! - Kevin Sweeney On May 2, 2015, 11:58 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33784/ --- (Updated May 2, 2015, 11:58 a.m.) Review request for Aurora and Kevin Sweeney. Bugs: AURORA-1204 https://issues.apache.org/jira/browse/AURORA-1204 Repository: aurora Description --- Symlink upstart configurations in vagrant rather than copying. Diffs - examples/vagrant/provision-dev-cluster.sh e7fdd4b7f5bdfcbe16f8ace1cef0dba00376aa81 Diff: https://reviews.apache.org/r/33784/diff/ Testing --- TODO(wfarner): Report back after verifying end-to-end tests pass for a fresh vagrant machine. Thanks, Bill Farner
Re: Review Request 33612: Add a task store implementation that uses a relational database.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33612/ --- (Updated May 5, 2015, 6:21 p.m.) Review request for Aurora, Joshua Cohen, Kevin Sweeney, and Maxim Khutornenko. Changes --- Fixed copy-paste error + rebase. Bugs: AURORA-556 https://issues.apache.org/jira/browse/AURORA-556 Repository: aurora Description --- Add a task store implementation that uses a relational database. The meat of this review is in `DbTaskStore`, `TaskConfigManager`, and the mapper xml files. Some supporting actors include files under views/, which serve DB business objects. I suggest reviewers start by skimming `DbTaskStore` and digesting the comments in there. There are some known loose ends here, most notably being continued performance enhancements and cleanup of relations in different tables. I've included several relevant TODOs, ~all of which must be addressed before this becomes the default task store. Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift 0182ecb3942cfa2d9dd21138779815f4500339be examples/vagrant/upstart/aurora-scheduler.conf cc4864c4d954d58e9bfaaaf5fc5afc8d0e032e78 src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java ed1171d52655fef643330f34913c256f77300fa2 src/main/java/org/apache/aurora/scheduler/app/SchedulerMain.java 3d19831ea0eb85032172b096ac87e126701aa218 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java 94ce5c3499ced1b63abf19787acc21b2cd4d0c75 src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java c8df88b77b9a2017c48bdc0c9a0477927ba0b179 src/main/java/org/apache/aurora/scheduler/storage/db/DbStorage.java 1a6c3f21d4fcb476539f588facecd8ef37332837 src/main/java/org/apache/aurora/scheduler/storage/db/DbTaskStore.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigManager.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/TaskMapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/ContainerShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/shims/TaskConstraintShim.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/ScheduleStatusTypeHandler.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/typehandlers/TypeHandlers.java 4d0c10d60037a3310226a6fd8936b84ae4135e7e src/main/java/org/apache/aurora/scheduler/storage/db/views/AssignedPort.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/ScheduledTaskWrapper.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskConfigRow.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/db/views/TaskLink.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/mem/InMemStoresModule.java 21f7d4db821930d2c5b52648e1996aa1ef12a85c src/main/resources/org/apache/aurora/scheduler/storage/db/JobUpdateDetailsMapper.xml f76f9a988669dab598e9d19f849018c6f55ce03e src/main/resources/org/apache/aurora/scheduler/storage/db/TaskConfigMapper.xml PRE-CREATION src/main/resources/org/apache/aurora/scheduler/storage/db/TaskMapper.xml PRE-CREATION src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java afb7db8eefa63b84d370877742870acdec58899c src/test/java/org/apache/aurora/scheduler/storage/AbstractTaskStoreTest.java e3b13407cb6875489e50cf93212845eab7aacb89 src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java PRE-CREATION src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java f18619a27eeb2aea8dcf01e54c23ed7d1c7d3d87 src/test/java/org/apache/aurora/scheduler/storage/mem/StorageTransactionTest.java bad9eb56b33c3e649c3b173e83d9c30da8f9317d Diff: https://reviews.apache.org/r/33612/diff/ Testing --- Unit tests and end-to-end tests, both using the new task store by default with this change. Thanks, Bill Farner
Re: Review Request 33783: Add stats for update state transitions.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33783/#review82550 --- Ship it! Ship It! - David McLaughlin On May 2, 2015, 6:41 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33783/ --- (Updated May 2, 2015, 6:41 p.m.) Review request for Aurora and David McLaughlin. Bugs: AURORA-1259 https://issues.apache.org/jira/browse/AURORA-1259 Repository: aurora Description --- This required moving `FakeStatsProvider` from src/test to src/main, since it's now used by `DbUtil`. I'm not totally happy about that, but i didn't see a more favorable approach. Diffs - config/legacy_untested_classes.txt 4d090a8fc1344531772e1b7025d77c2ff0a30035 src/main/java/org/apache/aurora/scheduler/storage/db/DBJobUpdateStore.java 94ce5c3499ced1b63abf19787acc21b2cd4d0c75 src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java a10a9e79c2e294d1f701e36a522e34222db1c3a3 src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 219676fab301542c129794416630070921225439 src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java f18619a27eeb2aea8dcf01e54c23ed7d1c7d3d87 src/test/java/org/apache/aurora/scheduler/testing/FakeStatsProvider.java Diff: https://reviews.apache.org/r/33783/diff/ Testing --- Thanks, Bill Farner
