Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70680 --- Ship it! Ship It! - Maxim Khutornenko On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
On Feb. 3, 2015, 12:48 a.m., Maxim Khutornenko wrote: The ticket suggests a possibility of the optimization route. Would you mind commenting why you decided to remove it after all? Bill Farner wrote: Sure. Kevin rightly pointed out that it's odd for us to check this _one_ invariant when there are many others. Put another way, we can't check everything, and doing this one check suggests that we are uncertain of our ability to maintain integrity of the data in this one specific way. Once we move this table to SQL, we will be in a much better position to continually enforce this type of constraint. SGTM. Thanks for explaining. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70675 --- On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30461: Adding pulse_interval_secs into client UpdateConfig.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/#review70662 --- src/main/python/apache/aurora/client/api/updater_util.py https://reviews.apache.org/r/30461/#comment115985 I suggest increasing this, say to 120. We already have a pretty good safety net built in, i wouldn't want to start battling micro-outages from the beginning. Also, a comment would be useful here to fill in context about what this is, what are the effects of a high/low value. src/main/python/apache/aurora/client/api/updater_util.py https://reviews.apache.org/r/30461/#comment115990 Can you investigate whether the python code respects the `isSetX` pattern? My hunch is that it does, and we should leverage it to distinguish default `0` from the user specifying `0`. src/main/python/apache/aurora/config/schema/base.py https://reviews.apache.org/r/30461/#comment115987 I think we should consider avoiding exposing this to end users. This setting is really a behind-the-scenes timeout that they probably lack the context to set appropriately. - Bill Farner On Jan. 30, 2015, 10:31 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/ --- (Updated Jan. 30, 2015, 10:31 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1071 https://issues.apache.org/jira/browse/AURORA-1071 Repository: aurora Description --- - Adding pulse_interval_secs into client UpdateConfig and validating its range - Raising an error in client updater for pulse_interval_secs Diffs - src/main/python/apache/aurora/client/api/updater.py 9f91de625f55514530a4f948d7ecdf7b5614b594 src/main/python/apache/aurora/client/api/updater_util.py 9d2e893a6ecff0fc48c7944575578443d41ced78 src/main/python/apache/aurora/config/schema/base.py e4433d2d47668f59bce169359131284d361bad09 src/test/python/apache/aurora/client/api/test_api.py ff1aff2eac391f219bc7c2483a16e35f916a224c src/test/python/apache/aurora/client/api/test_updater.py dd3f228c5062d388b4393aa4fd5b60a685bdb3a6 src/test/python/apache/aurora/client/api/test_updater_util.py fe3ac49491ca710761632405ac09de0cc0d038a5 Diff: https://reviews.apache.org/r/30461/diff/ Testing --- ./pants test.pytest src/test/python/apache/aurora/client:: Thanks, Maxim Khutornenko
Re: Review Request 30325: Implementing pulseJobUpdate RPC.
On Feb. 2, 2015, 4:16 p.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java, line 1535 https://reviews.apache.org/r/30325/diff/3/?file=841928#file841928line1535 This is too restrictive. It means that _only_ the update coordinator role may call this RPC, and that a user cannot build something to pulse their own updates. You should instead do what `isAdmin` does and fall back to the coordinator role if direct auth fails. This is an unfortunate state of affairs, and hopefully the move to shiro dramatically improves all this. Maxim Khutornenko wrote: I don't see how it's necessarily better. Pulsing can always be done under UPDATE_COORDINATOR membership, which is specifically covering heartbeats only. The isAdmin() requires ROOT that opens up everything else, including the ability to kill anyone's tasks in the claster. Isn't that more restrictive in real life? We don't expect regular users having ROOT access, meaning they will unlikely to get to write their own pulse routine due to that. Or maybe you suggesting an approach similar to killTasks(), where an admin check followed by a role validation? The problem here is that we don't have a job key to extract the role for authorization. Perhaps we can change the RPC to accept a job key instead but that would open up for a race we don't want to see (e.g. late pulse arrives for a wrong update ID). We could probably get away with both updateId and jobKey in the API to avoid ambiguity, or perhpas just updateId and role? I am open to suggestions. In Shiro this will look something like subject.checkPermission(update:resume:mesos:prod:labrat); With role update_coordinator having: ``` update:* ``` role root having: ``` * ``` and user mesos having: ``` *:*:mesos ``` So they'd all be allowed. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/#review70655 --- On Jan. 30, 2015, 9:23 a.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/ --- (Updated Jan. 30, 2015, 9:23 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1009 https://issues.apache.org/jira/browse/AURORA-1009 Repository: aurora Description --- Implemented the `pulseJobUpdate` RPC. Also, moved it into AuroraAdmin interface to support AOP capability validation. The RB is diffed against https://reviews.apache.org/r/30225/ Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 4f603f9e7ed004e53937a45ea8edf7241e15f5cf src/main/java/org/apache/aurora/auth/CapabilityValidator.java 45ef643ebe57c1517cdae373574331ea302a8b74 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 8c19f3b08135eb5f3098591ebf9931b42a086318 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 03d1fba76c23570c2c4102a48daf5ce035ecaaa3 Diff: https://reviews.apache.org/r/30325/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70675 --- The ticket suggests a possibility of the optimization route. Would you mind commenting why you decided to remove it after all? - Maxim Khutornenko On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30325: Implementing pulseJobUpdate RPC.
On Feb. 3, 2015, 12:16 a.m., Bill Farner wrote:
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java,
line 1535
https://reviews.apache.org/r/30325/diff/3/?file=841928#file841928line1535
This is too restrictive. It means that _only_ the update coordinator
role may call this RPC, and that a user cannot build something to pulse
their own updates.
You should instead do what `isAdmin` does and fall back to the
coordinator role if direct auth fails.
This is an unfortunate state of affairs, and hopefully the move to
shiro dramatically improves all this.
Maxim Khutornenko wrote:
I don't see how it's necessarily better. Pulsing can always be done under
UPDATE_COORDINATOR membership, which is specifically covering heartbeats
only. The isAdmin() requires ROOT that opens up everything else, including
the ability to kill anyone's tasks in the claster. Isn't that more
restrictive in real life? We don't expect regular users having ROOT access,
meaning they will unlikely to get to write their own pulse routine due to
that.
Or maybe you suggesting an approach similar to killTasks(), where an
admin check followed by a role validation? The problem here is that we don't
have a job key to extract the role for authorization. Perhaps we can change
the RPC to accept a job key instead but that would open up for a race we
don't want to see (e.g. late pulse arrives for a wrong update ID). We could
probably get away with both updateId and jobKey in the API to avoid
ambiguity, or perhpas just updateId and role? I am open to suggestions.
Kevin Sweeney wrote:
In Shiro this will look something like
subject.checkPermission(update:resume:mesos:prod:labrat);
With role update_coordinator having:
```
update:*
```
role root having:
```
*
```
and user mesos having:
```
*:*:mesos
```
So they'd all be allowed.
Or maybe you suggesting an approach similar to killTasks(), where an admin
check followed by a role validation?
Yeah, this. I'm not asking for update heartbeats to require ROOT.
The problem here is that we don't have a job key to extract the role for
authorization.
We have an association between {{updateId}} and role owning the job, right?
/**
* Fetches a read-only view of a job update.
*
* @param updateId Update ID to fetch.
* @return A read-only view of job update.
*/
OptionalIJobUpdate fetchJobUpdate(String updateId);
- Bill
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30325/#review70655
---
On Jan. 30, 2015, 5:23 p.m., Maxim Khutornenko wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30325/
---
(Updated Jan. 30, 2015, 5:23 p.m.)
Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner.
Bugs: AURORA-1009
https://issues.apache.org/jira/browse/AURORA-1009
Repository: aurora
Description
---
Implemented the `pulseJobUpdate` RPC. Also, moved it into AuroraAdmin
interface to support AOP capability validation.
The RB is diffed against https://reviews.apache.org/r/30225/
Diffs
-
api/src/main/thrift/org/apache/aurora/gen/api.thrift
4f603f9e7ed004e53937a45ea8edf7241e15f5cf
src/main/java/org/apache/aurora/auth/CapabilityValidator.java
45ef643ebe57c1517cdae373574331ea302a8b74
src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
8c19f3b08135eb5f3098591ebf9931b42a086318
src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
03d1fba76c23570c2c4102a48daf5ce035ecaaa3
Diff: https://reviews.apache.org/r/30325/diff/
Testing
---
./gradlew -Pq build
Thanks,
Maxim Khutornenko
Re: Review Request 30325: Implementing pulseJobUpdate RPC.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/#review70655 --- src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java https://reviews.apache.org/r/30325/#comment115972 You can check whether the primitive field is set, which will distinguish between default int value and an explicitly set value. I think it makes sense to reject zero at this layer. src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java https://reviews.apache.org/r/30325/#comment115976 This is too restrictive. It means that _only_ the update coordinator role may call this RPC, and that a user cannot build something to pulse their own updates. You should instead do what `isAdmin` does and fall back to the coordinator role if direct auth fails. This is an unfortunate state of affairs, and hopefully the move to shiro dramatically improves all this. - Bill Farner On Jan. 30, 2015, 5:23 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/ --- (Updated Jan. 30, 2015, 5:23 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1009 https://issues.apache.org/jira/browse/AURORA-1009 Repository: aurora Description --- Implemented the `pulseJobUpdate` RPC. Also, moved it into AuroraAdmin interface to support AOP capability validation. The RB is diffed against https://reviews.apache.org/r/30225/ Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 4f603f9e7ed004e53937a45ea8edf7241e15f5cf src/main/java/org/apache/aurora/auth/CapabilityValidator.java 45ef643ebe57c1517cdae373574331ea302a8b74 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 8c19f3b08135eb5f3098591ebf9931b42a086318 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 03d1fba76c23570c2c4102a48daf5ce035ecaaa3 Diff: https://reviews.apache.org/r/30325/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30325: Implementing pulseJobUpdate RPC.
On Feb. 3, 2015, 12:16 a.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java, line 1535 https://reviews.apache.org/r/30325/diff/3/?file=841928#file841928line1535 This is too restrictive. It means that _only_ the update coordinator role may call this RPC, and that a user cannot build something to pulse their own updates. You should instead do what `isAdmin` does and fall back to the coordinator role if direct auth fails. This is an unfortunate state of affairs, and hopefully the move to shiro dramatically improves all this. I don't see how it's necessarily better. Pulsing can always be done under UPDATE_COORDINATOR membership, which is specifically covering heartbeats only. The isAdmin() requires ROOT that opens up everything else, including the ability to kill anyone's tasks in the claster. Isn't that more restrictive in real life? We don't expect regular users having ROOT access, meaning they will unlikely to get to write their own pulse routine due to that. Or maybe you suggesting an approach similar to killTasks(), where an admin check followed by a role validation? The problem here is that we don't have a job key to extract the role for authorization. Perhaps we can change the RPC to accept a job key instead but that would open up for a race we don't want to see (e.g. late pulse arrives for a wrong update ID). We could probably get away with both updateId and jobKey in the API to avoid ambiguity, or perhpas just updateId and role? I am open to suggestions. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/#review70655 --- On Jan. 30, 2015, 5:23 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30325/ --- (Updated Jan. 30, 2015, 5:23 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1009 https://issues.apache.org/jira/browse/AURORA-1009 Repository: aurora Description --- Implemented the `pulseJobUpdate` RPC. Also, moved it into AuroraAdmin interface to support AOP capability validation. The RB is diffed against https://reviews.apache.org/r/30225/ Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift 4f603f9e7ed004e53937a45ea8edf7241e15f5cf src/main/java/org/apache/aurora/auth/CapabilityValidator.java 45ef643ebe57c1517cdae373574331ea302a8b74 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 8c19f3b08135eb5f3098591ebf9931b42a086318 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 03d1fba76c23570c2c4102a48daf5ce035ecaaa3 Diff: https://reviews.apache.org/r/30325/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim Khutornenko
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70671 --- Ship it! Ship It! - Kevin Sweeney On Feb. 2, 2015, 4:42 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 2, 2015, 4:42 p.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30461: Adding pulse_interval_secs into client UpdateConfig.
On Feb. 3, 2015, 12:28 a.m., Bill Farner wrote: src/main/python/apache/aurora/config/schema/base.py, line 35 https://reviews.apache.org/r/30461/diff/2/?file=842715#file842715line35 I think we should consider avoiding exposing this to end users. This setting is really a behind-the-scenes timeout that they probably lack the context to set appropriately. I take that back - this is a policy decision, and OSS end-users should have this knob. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/#review70662 --- On Jan. 30, 2015, 10:31 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/ --- (Updated Jan. 30, 2015, 10:31 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1071 https://issues.apache.org/jira/browse/AURORA-1071 Repository: aurora Description --- - Adding pulse_interval_secs into client UpdateConfig and validating its range - Raising an error in client updater for pulse_interval_secs Diffs - src/main/python/apache/aurora/client/api/updater.py 9f91de625f55514530a4f948d7ecdf7b5614b594 src/main/python/apache/aurora/client/api/updater_util.py 9d2e893a6ecff0fc48c7944575578443d41ced78 src/main/python/apache/aurora/config/schema/base.py e4433d2d47668f59bce169359131284d361bad09 src/test/python/apache/aurora/client/api/test_api.py ff1aff2eac391f219bc7c2483a16e35f916a224c src/test/python/apache/aurora/client/api/test_updater.py dd3f228c5062d388b4393aa4fd5b60a685bdb3a6 src/test/python/apache/aurora/client/api/test_updater_util.py fe3ac49491ca710761632405ac09de0cc0d038a5 Diff: https://reviews.apache.org/r/30461/diff/ Testing --- ./pants test.pytest src/test/python/apache/aurora/client:: Thanks, Maxim Khutornenko
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
On Feb. 3, 2015, 12:48 a.m., Maxim Khutornenko wrote: The ticket suggests a possibility of the optimization route. Would you mind commenting why you decided to remove it after all? Sure. Kevin rightly pointed out that it's odd for us to check this _one_ invariant when there are many others. Put another way, we can't check everything, and doing this one check suggests that we are uncertain of our ability to maintain integrity of the data in this one specific way. Once we move this table to SQL, we will be in a much better position to continually enforce this type of constraint. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70675 --- On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 29117: Adding thrift API changes document.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29117/#review70652 --- Ship it! Ship It! - Bill Farner On Jan. 21, 2015, 10:59 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29117/ --- (Updated Jan. 21, 2015, 10:59 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-973 https://issues.apache.org/jira/browse/AURORA-973 Repository: aurora Description --- This is a first stab at documenting thrift deprecation. Any suggestions/comments are welcome. Diffs - docs/developing-aurora-client.md b9912bce44d65ddd7f1e35f0ea9356a89d5fe767 docs/developing-aurora-scheduler.md 7f6cc2e6c8e01115a9b7a7dc7633bcd88ba02a0f docs/thrift-deprecation.md PRE-CREATION Diff: https://reviews.apache.org/r/29117/diff/ Testing --- https://github.com/maxim111333/incubator-aurora/blob/populated_deprecation/docs/thrift-deprecation.md Thanks, Maxim Khutornenko
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70684 --- Ship it! Master (a674581) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
Could the impact of this change be verified by our performance benchmarks? On Mon, Feb 2, 2015 at 5:06 PM, Aurora ReviewBot wfar...@apache.org wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70684 --- Ship it! Master (a674581) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner -- Zameer Manji
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
+1 to moving forward to this patch as-is On Mon, Feb 2, 2015 at 5:39 PM, Bill Farner wfar...@apache.org wrote: They could, but I don't think it would be worth the effort here. In this case, the code itself is of little value, it's a bonus that it is also a performance hog. On Monday, February 2, 2015, Zameer Manji zma...@twopensource.com wrote: Could the impact of this change be verified by our performance benchmarks? On Mon, Feb 2, 2015 at 5:06 PM, Aurora ReviewBot wfar...@apache.org wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70684 --- Ship it! Master (a674581) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner -- Zameer Manji -- -=Bill
Re: Review Request 30535: Remove shard uniqueness check from scheduler recovery phase.
They could, but I don't think it would be worth the effort here. In this case, the code itself is of little value, it's a bonus that it is also a performance hog. On Monday, February 2, 2015, Zameer Manji zma...@twopensource.com wrote: Could the impact of this change be verified by our performance benchmarks? On Mon, Feb 2, 2015 at 5:06 PM, Aurora ReviewBot wfar...@apache.org javascript:_e(%7B%7D,'cvml','wfar...@apache.org'); wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/#review70684 --- Ship it! Master (a674581) is green with this patch. ./build-support/jenkins/build.sh I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 3, 2015, 12:42 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30535/ --- (Updated Feb. 3, 2015, 12:42 a.m.) Review request for Aurora, David McLaughlin, Kevin Sweeney, and Maxim Khutornenko. Bugs: AURORA-1090 https://issues.apache.org/jira/browse/AURORA-1090 Repository: aurora Description --- Remove shard uniqueness check from scheduler recovery phase. Diffs - src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 1814658c044273f7c3a2348a16aea62e397cf860 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 93773eb5ba3bee1b3296e69ea30eabb531eeb661 Diff: https://reviews.apache.org/r/30535/diff/ Testing --- Thanks, Bill Farner -- Zameer Manji -- -=Bill
Re: Review Request 29171: Fix test_executor_vars so that it doesn't attempt to get a real PEX-INFO.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29171/#review70692 --- /nudge - Joe Smith On Dec. 17, 2014, 12:31 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29171/ --- (Updated Dec. 17, 2014, 12:31 p.m.) Review request for Aurora and Joe Smith. Repository: aurora Description --- Fix test_executor_vars so that it doesn't attempt to get a real PEX-INFO. Diffs - src/main/python/apache/aurora/executor/executor_vars.py 7c018271724ffab2ff6930e5802a48b50a39dded src/test/python/apache/aurora/executor/BUILD 3095f2a148b21365ea2f500039adb5710a187fa1 src/test/python/apache/aurora/executor/test_executor_vars.py af1c7b7d78b89ae713d6db96cf766b596fa91c55 Diff: https://reviews.apache.org/r/29171/diff/ Testing --- ./pants src/test/python/apache/aurora/executor:executor_vars -v Thanks, Brian Wickman
Re: Review Request 30461: Adding pulse_interval_secs into client UpdateConfig.
On Feb. 3, 2015, 12:28 a.m., Bill Farner wrote: src/main/python/apache/aurora/client/api/updater_util.py, line 25 https://reviews.apache.org/r/30461/diff/2/?file=842714#file842714line25 I suggest increasing this, say to 120. We already have a pretty good safety net built in, i wouldn't want to start battling micro-outages from the beginning. Also, a comment would be useful here to fill in context about what this is, what are the effects of a high/low value. This should probably be renamed to, say, MINUMUM_PULSE_INTERVAL_SECONDS? It doesn't seem to be used as a default anywhere that I see. Given that it's a minimum value, I think 120 seconds is probably too high. Maybe 60? - Joshua --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/#review70662 --- On Jan. 30, 2015, 10:31 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/ --- (Updated Jan. 30, 2015, 10:31 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1071 https://issues.apache.org/jira/browse/AURORA-1071 Repository: aurora Description --- - Adding pulse_interval_secs into client UpdateConfig and validating its range - Raising an error in client updater for pulse_interval_secs Diffs - src/main/python/apache/aurora/client/api/updater.py 9f91de625f55514530a4f948d7ecdf7b5614b594 src/main/python/apache/aurora/client/api/updater_util.py 9d2e893a6ecff0fc48c7944575578443d41ced78 src/main/python/apache/aurora/config/schema/base.py e4433d2d47668f59bce169359131284d361bad09 src/test/python/apache/aurora/client/api/test_api.py ff1aff2eac391f219bc7c2483a16e35f916a224c src/test/python/apache/aurora/client/api/test_updater.py dd3f228c5062d388b4393aa4fd5b60a685bdb3a6 src/test/python/apache/aurora/client/api/test_updater_util.py fe3ac49491ca710761632405ac09de0cc0d038a5 Diff: https://reviews.apache.org/r/30461/diff/ Testing --- ./pants test.pytest src/test/python/apache/aurora/client:: Thanks, Maxim Khutornenko
Re: Review Request 30461: Adding pulse_interval_secs into client UpdateConfig.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/#review70701 --- src/main/python/apache/aurora/client/api/updater.py https://reviews.apache.org/r/30461/#comment116052 s/in/by the src/test/python/apache/aurora/client/api/test_updater_util.py https://reviews.apache.org/r/30461/#comment116054 instead of hard coding to 10 here, set it to something like `UpdateConfig.DEFAULT_PULSE_INTERVAL_SECS - 1`? - Joshua Cohen On Jan. 30, 2015, 10:31 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30461/ --- (Updated Jan. 30, 2015, 10:31 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, and Bill Farner. Bugs: AURORA-1071 https://issues.apache.org/jira/browse/AURORA-1071 Repository: aurora Description --- - Adding pulse_interval_secs into client UpdateConfig and validating its range - Raising an error in client updater for pulse_interval_secs Diffs - src/main/python/apache/aurora/client/api/updater.py 9f91de625f55514530a4f948d7ecdf7b5614b594 src/main/python/apache/aurora/client/api/updater_util.py 9d2e893a6ecff0fc48c7944575578443d41ced78 src/main/python/apache/aurora/config/schema/base.py e4433d2d47668f59bce169359131284d361bad09 src/test/python/apache/aurora/client/api/test_api.py ff1aff2eac391f219bc7c2483a16e35f916a224c src/test/python/apache/aurora/client/api/test_updater.py dd3f228c5062d388b4393aa4fd5b60a685bdb3a6 src/test/python/apache/aurora/client/api/test_updater_util.py fe3ac49491ca710761632405ac09de0cc0d038a5 Diff: https://reviews.apache.org/r/30461/diff/ Testing --- ./pants test.pytest src/test/python/apache/aurora/client:: Thanks, Maxim Khutornenko
Re: Review Request 30467: Update mesos lib to 0.21.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30467/#review70593 --- Master (9233abd) is red with this patch. ./build-support/jenkins/build.sh :buildSrc:compileJava UP-TO-DATE :buildSrc:processResources UP-TO-DATE :buildSrc:classes UP-TO-DATE :buildSrc:jar :buildSrc:assemble :buildSrc:compileTestJava UP-TO-DATE :buildSrc:processTestResources UP-TO-DATE :buildSrc:testClasses UP-TO-DATE :buildSrc:test UP-TO-DATE :buildSrc:check UP-TO-DATE :buildSrc:build BUILD SUCCESSFUL Total time: 3 mins 44.917 secs + export PIP_DEFAULT_TIMEOUT=60 + PIP_DEFAULT_TIMEOUT=60 + mkdir -p third_party + pip install -d third_party -r /dev/fd/63 ++ grep -v mesos.native 3rdparty/python/requirements.txt Downloading/unpacking bottle==0.11.6 (from -r /dev/fd/63 (line 1)) Saved ./third_party/bottle-0.11.6-py2.py3-none-any.whl Downloading/unpacking CherryPy==3.6.0 (from -r /dev/fd/63 (line 2)) Saved ./third_party/CherryPy-3.6.0.tar.gz Running setup.py (path:/tmp/user/2396/pip_build_jenkins/CherryPy/setup.py) egg_info for package CherryPy Downloading/unpacking mako==0.4.0 (from -r /dev/fd/63 (line 3)) Saved ./third_party/Mako-0.4.0.tar.gz Running setup.py (path:/tmp/user/2396/pip_build_jenkins/mako/setup.py) egg_info for package mako warning: no files found matching '*.xml' under directory 'examples' warning: no files found matching '*.mako' under directory 'examples' warning: no files found matching 'ez_setup.py' no previously-included directories found matching 'doc/build/output' Downloading/unpacking mesos.interface==0.21.1 (from -r /dev/fd/63 (line 4)) Could not find a version that satisfies the requirement mesos.interface==0.21.1 (from -r /dev/fd/63 (line 4)) (from versions: 0.20.0, 0.20.1, 0.21.0) Some externally hosted files were ignored (use --allow-external to allow). Cleaning up... No distributions matching the version for mesos.interface==0.21.1 (from -r /dev/fd/63 (line 4)) Storing debug log for failure in /home/jenkins/.pip/pip.log I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 2, 2015, 6:10 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30467/ --- (Updated Feb. 2, 2015, 6:10 p.m.) Review request for Aurora and Kevin Sweeney. Bugs: AURORA-1080 https://issues.apache.org/jira/browse/AURORA-1080 Repository: aurora Description --- This is needed to take advantage of the task reconciliation API and new status update fields. Diffs - 3rdparty/python/requirements.txt 0c88e4c299ffe5886bb3ab7d83b0187cb4b7888a build-support/python/make-mesos-native-egg 2cba8ede04e22aee1e6ec5c979161904a6f8fddb build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 docs/deploying-aurora-scheduler.md 9bf5b5a92bf65b31b2f8fda102003b113f61186a examples/vagrant/provision-dev-cluster.sh fa0de88314169bc9fe31aaf41126cedd0865ed5a Diff: https://reviews.apache.org/r/30467/diff/ Testing --- This is blocked by https://issues.apache.org/jira/browse/MESOS-2310 Thanks, Bill Farner
Re: Review Request 30467: Update mesos lib to 0.21.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30467/ --- (Updated Feb. 2, 2015, 6:10 p.m.) Review request for Aurora and Kevin Sweeney. Changes --- updated to fix centos7 build deps. native eggs are now posted, still blocked by MESOS-2310 though. Bugs: AURORA-1080 https://issues.apache.org/jira/browse/AURORA-1080 Repository: aurora Description --- This is needed to take advantage of the task reconciliation API and new status update fields. Diffs (updated) - 3rdparty/python/requirements.txt 0c88e4c299ffe5886bb3ab7d83b0187cb4b7888a build-support/python/make-mesos-native-egg 2cba8ede04e22aee1e6ec5c979161904a6f8fddb build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 docs/deploying-aurora-scheduler.md 9bf5b5a92bf65b31b2f8fda102003b113f61186a examples/vagrant/provision-dev-cluster.sh fa0de88314169bc9fe31aaf41126cedd0865ed5a Diff: https://reviews.apache.org/r/30467/diff/ Testing --- This is blocked by https://issues.apache.org/jira/browse/MESOS-2310 Thanks, Bill Farner
Re: Review Request 28731: Implemented TaskScheduler benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/#review70591 --- Master (9233abd) is green with this patch. ./build-support/jenkins/build.sh However, it appears that it might lack test coverage. I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 2, 2015, 5:39 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/ --- (Updated Feb. 2, 2015, 5:39 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-969 https://issues.apache.org/jira/browse/AURORA-969 Repository: aurora Description --- Added baseline benchmarks for a few static veto cases. Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java 03c2a8fde4a3fe5ac73f44da2cbe227995501c46 src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 10c4f06e3fde91d5ed80da3381caa9a3a6a0af33 Diff: https://reviews.apache.org/r/28731/diff/ Testing --- Sample run on a local box: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 10 iterations, 1 s each # Measurement: 100 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark # Run progress: 0.00% complete, ETA 00:05:30 # Fork: 1 of 1 # Warmup Iteration 1: 278284250.000 ns/op # Warmup Iteration 2: 70294312.500 ns/op # Warmup Iteration 3: 19293379.310 ns/op # Warmup Iteration 4: 11945387.097 ns/op # Warmup Iteration 5: 10725388.350 ns/op # Warmup Iteration 6: 13043282.353 ns/op # Warmup Iteration 7: 9233083.333 ns/op # Warmup Iteration 8: 9521051.724 ns/op # Warmup Iteration 9: 10750854.369 ns/op # Warmup Iteration 10: 7460243.243 ns/op Iteration 1: 7885364.286 ns/op Iteration 2: 7735139.860 ns/op Iteration 3: 7660208.333 ns/op Iteration 4: 7761204.225 ns/op Iteration 5: 7868907.143 ns/op Iteration 6: 7567404.110 ns/op Iteration 7: 7611000.000 ns/op Iteration 8: 7766154.930 ns/op Iteration 9: 7669344.828 ns/op Iteration 10: 7707783.217 ns/op Iteration 11: 7435651.007 ns/op Iteration 12: 7697631.944 ns/op Iteration 13: 7712531.469 ns/op Iteration 14: 7899407.143 ns/op Iteration 15: 7448472.973 ns/op Iteration 16: 7791521.127 ns/op Iteration 17: 7612213.793 ns/op Iteration 18: 7710867.133 ns/op Iteration 19: 7649296.552 ns/op Iteration 20: 7768309.859 ns/op Iteration 21: 7688666.667 ns/op Iteration 22: 7531557.823 ns/op Iteration 23: 7381193.333 ns/op Iteration 24: 7726006.993 ns/op Iteration 25: 7603358.621 ns/op Iteration 26: 7653631.944 ns/op Iteration 27: 7442275.168 ns/op Iteration 28: 7613186.207 ns/op Iteration 29: 7765823.944 ns/op Iteration 30: 7489687.075 ns/op Iteration 31: 7811443.662 ns/op Iteration 32: 8015007.246 ns/op Iteration 33: 8192392.593 ns/op Iteration 34: 8040335.766 ns/op Iteration 35: 7584212.329 ns/op Iteration 36: 8001934.783 ns/op Iteration 37: 9744815.789 ns/op Iteration 38: 11688284.211 ns/op Iteration 39: 8661406.250 ns/op Iteration 40: 7678413.793 ns/op Iteration 41: 8502223.077 ns/op Iteration 42: 7640820.690 ns/op Iteration 43: 7875624.113 ns/op Iteration 44: 7506809.524 ns/op Iteration 45: 8005431.655 ns/op Iteration 46: 8081664.234 ns/op Iteration 47: 7579438.356 ns/op Iteration 48:
Re: Review Request 30467: Update mesos lib to 0.21.1
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30467/#review70592 --- I'll hold off on reviewing this until it's possible to build. - Kevin Sweeney On Feb. 2, 2015, 10:10 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30467/ --- (Updated Feb. 2, 2015, 10:10 a.m.) Review request for Aurora and Kevin Sweeney. Bugs: AURORA-1080 https://issues.apache.org/jira/browse/AURORA-1080 Repository: aurora Description --- This is needed to take advantage of the task reconciliation API and new status update fields. Diffs - 3rdparty/python/requirements.txt 0c88e4c299ffe5886bb3ab7d83b0187cb4b7888a build-support/python/make-mesos-native-egg 2cba8ede04e22aee1e6ec5c979161904a6f8fddb build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 docs/deploying-aurora-scheduler.md 9bf5b5a92bf65b31b2f8fda102003b113f61186a examples/vagrant/provision-dev-cluster.sh fa0de88314169bc9fe31aaf41126cedd0865ed5a Diff: https://reviews.apache.org/r/30467/diff/ Testing --- This is blocked by https://issues.apache.org/jira/browse/MESOS-2310 Thanks, Bill Farner
Re: Review Request 30480: Use checkstyle 6.2.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30480/#review70584 --- Ship it! Ship It! - Maxim Khutornenko On Jan. 31, 2015, 8:31 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30480/ --- (Updated Jan. 31, 2015, 8:31 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1081 https://issues.apache.org/jira/browse/AURORA-1081 Repository: aurora Description --- The configuration changes reflect features removed from checkstyle: https://github.com/checkstyle/checkstyle/pull/209 https://github.com/checkstyle/checkstyle/issues/457 Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 config/checkstyle/checkstyle.xml 20709896213b56c0dbdeaf790c826839383df20b Diff: https://reviews.apache.org/r/30480/diff/ Testing --- Thanks, Bill Farner
Re: Review Request 28731: Implemented TaskScheduler benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/ --- (Updated Feb. 2, 2015, 5:39 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Changes --- Rebasing. Bugs: AURORA-969 https://issues.apache.org/jira/browse/AURORA-969 Repository: aurora Description --- Added baseline benchmarks for a few static veto cases. Diffs (updated) - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java 03c2a8fde4a3fe5ac73f44da2cbe227995501c46 src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 10c4f06e3fde91d5ed80da3381caa9a3a6a0af33 Diff: https://reviews.apache.org/r/28731/diff/ Testing --- Sample run on a local box: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 10 iterations, 1 s each # Measurement: 100 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark # Run progress: 0.00% complete, ETA 00:05:30 # Fork: 1 of 1 # Warmup Iteration 1: 278284250.000 ns/op # Warmup Iteration 2: 70294312.500 ns/op # Warmup Iteration 3: 19293379.310 ns/op # Warmup Iteration 4: 11945387.097 ns/op # Warmup Iteration 5: 10725388.350 ns/op # Warmup Iteration 6: 13043282.353 ns/op # Warmup Iteration 7: 9233083.333 ns/op # Warmup Iteration 8: 9521051.724 ns/op # Warmup Iteration 9: 10750854.369 ns/op # Warmup Iteration 10: 7460243.243 ns/op Iteration 1: 7885364.286 ns/op Iteration 2: 7735139.860 ns/op Iteration 3: 7660208.333 ns/op Iteration 4: 7761204.225 ns/op Iteration 5: 7868907.143 ns/op Iteration 6: 7567404.110 ns/op Iteration 7: 7611000.000 ns/op Iteration 8: 7766154.930 ns/op Iteration 9: 7669344.828 ns/op Iteration 10: 7707783.217 ns/op Iteration 11: 7435651.007 ns/op Iteration 12: 7697631.944 ns/op Iteration 13: 7712531.469 ns/op Iteration 14: 7899407.143 ns/op Iteration 15: 7448472.973 ns/op Iteration 16: 7791521.127 ns/op Iteration 17: 7612213.793 ns/op Iteration 18: 7710867.133 ns/op Iteration 19: 7649296.552 ns/op Iteration 20: 7768309.859 ns/op Iteration 21: 7688666.667 ns/op Iteration 22: 7531557.823 ns/op Iteration 23: 7381193.333 ns/op Iteration 24: 7726006.993 ns/op Iteration 25: 7603358.621 ns/op Iteration 26: 7653631.944 ns/op Iteration 27: 7442275.168 ns/op Iteration 28: 7613186.207 ns/op Iteration 29: 7765823.944 ns/op Iteration 30: 7489687.075 ns/op Iteration 31: 7811443.662 ns/op Iteration 32: 8015007.246 ns/op Iteration 33: 8192392.593 ns/op Iteration 34: 8040335.766 ns/op Iteration 35: 7584212.329 ns/op Iteration 36: 8001934.783 ns/op Iteration 37: 9744815.789 ns/op Iteration 38: 11688284.211 ns/op Iteration 39: 8661406.250 ns/op Iteration 40: 7678413.793 ns/op Iteration 41: 8502223.077 ns/op Iteration 42: 7640820.690 ns/op Iteration 43: 7875624.113 ns/op Iteration 44: 7506809.524 ns/op Iteration 45: 8005431.655 ns/op Iteration 46: 8081664.234 ns/op Iteration 47: 7579438.356 ns/op Iteration 48: 7993405.797 ns/op Iteration 49: 7571958.904 ns/op Iteration 50: 8116463.235 ns/op Iteration 51: 7941330.935 ns/op Iteration 52: 7687145.833 ns/op Iteration 53: 8082554.745 ns/op Iteration 54: 7597889.655 ns/op Iteration 55: 7299907.285 ns/op Iteration 56: 7992789.855 ns/op Iteration 57: 7648268.966 ns/op Iteration 58: 7570863.014 ns/op Iteration 59: 7885078.571 ns/op Iteration 60: 7647158.621 ns/op Iteration 61: 7830858.156 ns/op Iteration 62: 7773690.141 ns/op Iteration 63: 7905850.000 ns/op Iteration 64: 7653800.000 ns/op Iteration 65: 7408248.322 ns/op Iteration 66: 7961352.518 ns/op
Re: Review Request 28731: Implemented TaskScheduler benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/#review70587 --- @ReviewBot retry - Maxim Khutornenko On Feb. 2, 2015, 5:39 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/ --- (Updated Feb. 2, 2015, 5:39 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-969 https://issues.apache.org/jira/browse/AURORA-969 Repository: aurora Description --- Added baseline benchmarks for a few static veto cases. Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java 03c2a8fde4a3fe5ac73f44da2cbe227995501c46 src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 10c4f06e3fde91d5ed80da3381caa9a3a6a0af33 Diff: https://reviews.apache.org/r/28731/diff/ Testing --- Sample run on a local box: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 10 iterations, 1 s each # Measurement: 100 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark # Run progress: 0.00% complete, ETA 00:05:30 # Fork: 1 of 1 # Warmup Iteration 1: 278284250.000 ns/op # Warmup Iteration 2: 70294312.500 ns/op # Warmup Iteration 3: 19293379.310 ns/op # Warmup Iteration 4: 11945387.097 ns/op # Warmup Iteration 5: 10725388.350 ns/op # Warmup Iteration 6: 13043282.353 ns/op # Warmup Iteration 7: 9233083.333 ns/op # Warmup Iteration 8: 9521051.724 ns/op # Warmup Iteration 9: 10750854.369 ns/op # Warmup Iteration 10: 7460243.243 ns/op Iteration 1: 7885364.286 ns/op Iteration 2: 7735139.860 ns/op Iteration 3: 7660208.333 ns/op Iteration 4: 7761204.225 ns/op Iteration 5: 7868907.143 ns/op Iteration 6: 7567404.110 ns/op Iteration 7: 7611000.000 ns/op Iteration 8: 7766154.930 ns/op Iteration 9: 7669344.828 ns/op Iteration 10: 7707783.217 ns/op Iteration 11: 7435651.007 ns/op Iteration 12: 7697631.944 ns/op Iteration 13: 7712531.469 ns/op Iteration 14: 7899407.143 ns/op Iteration 15: 7448472.973 ns/op Iteration 16: 7791521.127 ns/op Iteration 17: 7612213.793 ns/op Iteration 18: 7710867.133 ns/op Iteration 19: 7649296.552 ns/op Iteration 20: 7768309.859 ns/op Iteration 21: 7688666.667 ns/op Iteration 22: 7531557.823 ns/op Iteration 23: 7381193.333 ns/op Iteration 24: 7726006.993 ns/op Iteration 25: 7603358.621 ns/op Iteration 26: 7653631.944 ns/op Iteration 27: 7442275.168 ns/op Iteration 28: 7613186.207 ns/op Iteration 29: 7765823.944 ns/op Iteration 30: 7489687.075 ns/op Iteration 31: 7811443.662 ns/op Iteration 32: 8015007.246 ns/op Iteration 33: 8192392.593 ns/op Iteration 34: 8040335.766 ns/op Iteration 35: 7584212.329 ns/op Iteration 36: 8001934.783 ns/op Iteration 37: 9744815.789 ns/op Iteration 38: 11688284.211 ns/op Iteration 39: 8661406.250 ns/op Iteration 40: 7678413.793 ns/op Iteration 41: 8502223.077 ns/op Iteration 42: 7640820.690 ns/op Iteration 43: 7875624.113 ns/op Iteration 44: 7506809.524 ns/op Iteration 45: 8005431.655 ns/op Iteration 46: 8081664.234 ns/op Iteration 47: 7579438.356 ns/op Iteration 48: 7993405.797 ns/op Iteration 49: 7571958.904 ns/op Iteration 50: 8116463.235 ns/op Iteration 51: 7941330.935 ns/op Iteration 52: 7687145.833 ns/op Iteration 53: 8082554.745 ns/op Iteration
Re: Review Request 28731: Implemented TaskScheduler benchmarks.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/#review70586 --- Master (9233abd) is red with this patch. ./build-support/jenkins/build.sh :checkstyleJmh :jsHint :checkstyleMain :compileTestJava :processTestResources :testClasses :checkstyleTest :findbugsJmh :findbugsMain :findbugsTest :licenseJmh UP-TO-DATE :licenseMain UP-TO-DATE :licenseTest UP-TO-DATE :license UP-TO-DATE :pmdMain :test org.apache.aurora.scheduler.http.ServletFilterTest testGzipEncoding FAILED java.lang.AssertionError at ServletFilterTest.java:49 774 tests completed, 1 failed, 1 skipped :test FAILED :jacocoTestReport Coverage report generated: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/jacoco/test/html/index.html :analyzeReport Instruction coverage of 0.891844551356906 exceeds minimum coverage of 0.89. Branch coverage of 0.8365580448065173 exceeds minimum coverage of 0.835. FAILURE: Build failed with an exception. * What went wrong: Execution failed for task ':test'. There were failing tests. See the report at: file:///home/jenkins/jenkins-slave/workspace/AuroraBot/dist/reports/tests/index.html * Try: Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. BUILD FAILED Total time: 3 mins 55.251 secs I will refresh this build result if you post a review containing @ReviewBot retry - Aurora ReviewBot On Feb. 2, 2015, 5:39 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28731/ --- (Updated Feb. 2, 2015, 5:39 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: AURORA-969 https://issues.apache.org/jira/browse/AURORA-969 Repository: aurora Description --- Added baseline benchmarks for a few static veto cases. Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 src/jmh/java/org/apache/aurora/benchmark/Hosts.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Offers.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/SchedulerBenchmark.java 5cecada93e4e04b689e826af49f691ed7e94ae49 src/jmh/java/org/apache/aurora/benchmark/SchedulingBenchmarks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/Tasks.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeDriver.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeRescheduleCalculator.java PRE-CREATION src/jmh/java/org/apache/aurora/benchmark/fakes/FakeStatsProvider.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/async/TaskScheduler.java b6402ae42e3c7e4dca1c120fa6ef82d2d69e69d5 src/main/java/org/apache/aurora/scheduler/async/preemptor/CachedClusterState.java 03c2a8fde4a3fe5ac73f44da2cbe227995501c46 src/main/java/org/apache/aurora/scheduler/async/preemptor/ClusterState.java f7e157c890b5627411acd4bd5c2559ef4829147c src/main/java/org/apache/aurora/scheduler/async/preemptor/PreemptorImpl.java 10c4f06e3fde91d5ed80da3381caa9a3a6a0af33 Diff: https://reviews.apache.org/r/28731/diff/ Testing --- Sample run on a local box: ``` # VM invoker: /Library/Java/JavaVirtualMachines/jdk1.7.0_25.jdk/Contents/Home/jre/bin/java # VM options: -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant # Warmup: 10 iterations, 1 s each # Measurement: 100 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 1 thread, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: org.apache.aurora.benchmark.SchedulingBenchmarks.ConstraintMismatchsSchedulingBenchmark.runBenchmark # Run progress: 0.00% complete, ETA 00:05:30 # Fork: 1 of 1 # Warmup Iteration 1: 278284250.000 ns/op # Warmup Iteration 2: 70294312.500 ns/op # Warmup Iteration 3: 19293379.310 ns/op # Warmup Iteration 4: 11945387.097 ns/op # Warmup Iteration 5: 10725388.350 ns/op # Warmup Iteration 6: 13043282.353 ns/op # Warmup Iteration 7: 9233083.333 ns/op # Warmup Iteration 8: 9521051.724 ns/op # Warmup Iteration 9: 10750854.369 ns/op # Warmup Iteration 10: 7460243.243 ns/op Iteration 1: 7885364.286 ns/op Iteration 2: 7735139.860 ns/op Iteration 3: 7660208.333 ns/op Iteration 4: 7761204.225 ns/op Iteration 5: 7868907.143 ns/op Iteration 6: 7567404.110 ns/op Iteration 7: 7611000.000 ns/op Iteration 8: 7766154.930 ns/op Iteration 9: 7669344.828 ns/op Iteration 10: 7707783.217 ns/op Iteration 11: 7435651.007 ns/op Iteration 12: 7697631.944 ns/op Iteration 13: 7712531.469 ns/op Iteration 14: 7899407.143 ns/op Iteration 15: 7448472.973 ns/op Iteration
Re: Review Request 30480: Use checkstyle 6.2.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30480/#review70590 --- Ship it! Ship It! - Kevin Sweeney On Jan. 31, 2015, 12:31 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30480/ --- (Updated Jan. 31, 2015, 12:31 p.m.) Review request for Aurora and Maxim Khutornenko. Bugs: AURORA-1081 https://issues.apache.org/jira/browse/AURORA-1081 Repository: aurora Description --- The configuration changes reflect features removed from checkstyle: https://github.com/checkstyle/checkstyle/pull/209 https://github.com/checkstyle/checkstyle/issues/457 Diffs - build.gradle f9f71a84493b782e9f6072e44e89a2c017cf2a09 config/checkstyle/checkstyle.xml 20709896213b56c0dbdeaf790c826839383df20b Diff: https://reviews.apache.org/r/30480/diff/ Testing --- Thanks, Bill Farner
