Review Request 26753: Start removing clientv1.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26753/ --- Review request for Aurora, Bill Farner and Zameer Manji. Bugs: aurora-131 https://issues.apache.org/jira/browse/aurora-131 Repository: aurora Description --- - Make the aurora2 main target be the standalone clientv2; - Don't build the bridged client by default; but allow forced build using aurora2_bridge. - Modify cli tests so that they only depend on the standalone clientv2. - Modify vagrant config to make clientv2 the primary client. - Modify end-to-end tests to match the vagrant changes. (Note: end-to-end tests are failing, but in the same way as they fail without this change. I'm working on debugging that, but it should be its own change/review, and I don't want to delay reviewing this.) Diffs - docs/developing-aurora-client.md e1b2ccd7504f983169118a288721894184d67c97 examples/vagrant/aurorabuild.sh a27636655d722ca79f66b377fd847954d52e8feb examples/vagrant/provision-dev-cluster.sh 740bc212ba604b2c64af92eba1be41e8ed3fdbde src/main/python/apache/aurora/client/cli/BUILD 995570325bbb09ecbcc2ace5d223760c5d49367f src/test/python/apache/aurora/client/cli/test_api_from_cli.py a2b28ba23961284ba60358af54726e0386dd69b6 src/test/python/apache/aurora/client/cli/test_cancel_update.py e7052465411165acb3d5145664f2f166ac052500 src/test/python/apache/aurora/client/cli/test_command_hooks.py 9fc6fe2c2063cda494437d83044557b345acacea src/test/python/apache/aurora/client/cli/test_config_noun.py dfcbd7217b1d51609fa01c4d9cefed5471c91718 src/test/python/apache/aurora/client/cli/test_create.py 427f7ce4476b48d407b8bd2bf2c54c52e6e63079 src/test/python/apache/aurora/client/cli/test_cron.py c7b71c29d44150162fec8066947623fa91815424 src/test/python/apache/aurora/client/cli/test_diff.py 10817695352687cdb5b0c3ed9720e3091b230e68 src/test/python/apache/aurora/client/cli/test_help.py f73c8a3778b7d118ea2865f213b442a607fb4a7d src/test/python/apache/aurora/client/cli/test_inspect.py e997b9743b63d71f8624ecf5ca1dcae0227be70d src/test/python/apache/aurora/client/cli/test_kill.py bac4485fa105848d96e2505c4a2ea2eee45dc968 src/test/python/apache/aurora/client/cli/test_logging.py 9ca4dceeaa87d5fb2e38fe0d83fdcdf1ee597a0e src/test/python/apache/aurora/client/cli/test_open.py c20649f5cada241d0f6e9ae5f88d300eac073517 src/test/python/apache/aurora/client/cli/test_plugins.py dc5edd4f03cee062673231a04908193480c8071c src/test/python/apache/aurora/client/cli/test_quota.py 88fb9aec4d1eae6ad05da01752a670f902bafb1b src/test/python/apache/aurora/client/cli/test_restart.py a5f94484b30ecb8417116db9ce12c015957357c5 src/test/python/apache/aurora/client/cli/test_sla.py a1a3d8161ba747aa23a5e614e9ae31473d2058c1 src/test/python/apache/aurora/client/cli/test_status.py c704daec5a6eee73c7092a201b168881853908e8 src/test/python/apache/aurora/client/cli/test_supdate.py 6775c389cb1a0b80dd17fe179e8b98d4e9db0332 src/test/python/apache/aurora/client/cli/test_task_run.py 16fde14c03f6fd2c000e76625fad174835763f1b src/test/python/apache/aurora/client/cli/test_update.py cff1b6578aec6f5bcc1e610e58b47af233f32b41 src/test/sh/org/apache/aurora/e2e/test_common.sh 43d2516133c6d6cdb4236358f942396f057f739c src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 324aa4dbeff00e673fe73b87e3a0766856cd213c src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh 8f4d2b01c9fa5a6ec9e8885a2d4fa0e9c3abb8a1 Diff: https://reviews.apache.org/r/26753/diff/ Testing --- - Ran all unit tests, confirmed they continue to pass. - Ran end-to-end tests (both v1 and v2 variants) with the updated setup, and verified that they fail in exactly the same way as before this change. Thanks, Mark Chu-Carroll
Re: Review Request 26720: Limit the amount of work done while bootstrapping thrift.
On Oct. 15, 2014, 1:06 a.m., Kevin Sweeney wrote: build-support/thrift/Makefile, line 49 https://reviews.apache.org/r/26720/diff/1/?file=721229#file721229line49 Not your issue but this line appears impossible to execute. Removed. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26720/#review56632 --- On Oct. 15, 2014, 12:59 a.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26720/ --- (Updated Oct. 15, 2014, 12:59 a.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Repository: aurora Description --- Limit the amount of work done while bootstrapping thrift. Diffs - build-support/thrift/Makefile 2b62b15ee946862b2bea6a4df2b7fdec2be1426a Diff: https://reviews.apache.org/r/26720/diff/ Testing --- Through a single trial on my workstation: Before this patch: $ git clean -fdx $ time ./gradlew build /dev/null real 2m2.406s user 5m14.128s sys 0m12.222s After this patch: $ git clean -fdx $ time ./gradlew build /dev/null real 1m30.926s user 5m18.393s sys 0m11.148s Thanks, Bill Farner
Re: Review Request 26720: Limit the amount of work done while bootstrapping thrift.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26720/ --- (Updated Oct. 15, 2014, 4:22 p.m.) Review request for Aurora, Kevin Sweeney and Zameer Manji. Repository: aurora Description --- Limit the amount of work done while bootstrapping thrift. Diffs (updated) - build-support/thrift/Makefile 2b62b15ee946862b2bea6a4df2b7fdec2be1426a Diff: https://reviews.apache.org/r/26720/diff/ Testing --- Through a single trial on my workstation: Before this patch: $ git clean -fdx $ time ./gradlew build /dev/null real2m2.406s user5m14.128s sys 0m12.222s After this patch: $ git clean -fdx $ time ./gradlew build /dev/null real1m30.926s user5m18.393s sys 0m11.148s Thanks, Bill Farner
Re: Review Request 26688: Fix errors in help rendering:
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26688/ --- (Updated Oct. 15, 2014, 12:41 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: aurora-831 https://issues.apache.org/jira/browse/aurora-831 Repository: aurora Description --- - Put plugin-generated options into the correct order. - Include the option-name in the detailed help list. - Add missing metavars. Diffs - src/main/python/apache/aurora/client/cli/__init__.py da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8 src/main/python/apache/aurora/client/cli/options.py dc76c25b90acb9610e40b939e65c3cabf032649f src/main/python/apache/aurora/client/cli/standalone_client.py 20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1 src/test/python/apache/aurora/client/cli/test_help.py f73c8a3778b7d118ea2865f213b442a607fb4a7d Diff: https://reviews.apache.org/r/26688/diff/ Testing --- Thanks, Mark Chu-Carroll
Re: Review Request 26688: Fix errors in help rendering:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/
---
(Updated Oct. 15, 2014, 12:46 p.m.)
Review request for Aurora, Joshua Cohen and Zameer Manji.
Bugs: aurora-831
https://issues.apache.org/jira/browse/aurora-831
Repository: aurora
Description (updated)
---
- Put plugin-generated options into the correct order.
- Include the option-name in the detailed help list.
- Add missing metavars.
Diffs
-
src/main/python/apache/aurora/client/cli/__init__.py
da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8
src/main/python/apache/aurora/client/cli/options.py
dc76c25b90acb9610e40b939e65c3cabf032649f
src/main/python/apache/aurora/client/cli/standalone_client.py
20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1
src/test/python/apache/aurora/client/cli/test_help.py
f73c8a3778b7d118ea2865f213b442a607fb4a7d
Diff: https://reviews.apache.org/r/26688/diff/
Testing (updated)
---
New help output:
{noformat}
[sun-wukong incubator-aurora (deprecate)]$ ./dist/aurora2.pex help cron
deschedule
Usage for verb cron deschedule:
deschedule [--verbose-logging] [--logging-level=numeric_level]
[--error-log-dir=error-log-dir] [--bind=var=value] CLUSTER/ROLE/ENV/NAME
Options:
--bind=var=value
Bind a pystachio variable name to a value. Multiple flags may be used
to specify multiple values.
CLUSTER/ROLE/ENV/NAME
Fully specified job key, in CLUSTER/ROLE/ENV/NAME format
--verbose-logging
Show verbose logging, including all logs up to level INFO (equivalent
to --logging-level=20)
--logging-level=numeric_level
Set logging to a specific numeric level.
--error-log-dir=error-log-dir
Directory location where error files containing stack traces should be
written. If the directory doesn't exist, it will be created
Remove the cron schedule for a job.
{noformat}
Thanks,
Mark Chu-Carroll
Re: Review Request 26688: Fix errors in help rendering:
On Oct. 14, 2014, 12:46 p.m., Joshua Cohen wrote: src/test/python/apache/aurora/client/cli/test_help.py, line 75 https://reviews.apache.org/r/26688/diff/1/?file=720844#file720844line75 Are option names guaranteed to be unique? If not this test could potentially pass if any help output contains a plugin option name, not necessarily the help output for the command to which the plugin was registered. It's also possible for a plugin option name to appear in the help for another option, and not on its own, which would cause this test to succeed even if the plugin options themselves are not properly displayed? I guess what I'm getting at is would it be better to test for more than just the appearance of a string at any point in the output? (This may be based on incomplete understanding of how the client registers commands/options). I'm trying to keep the test from being overly sensitive to changes. The problem with a lot of output testing is that it's incredibly brittle - even tiny changes to the output require the test to get rewritten. So when possible, I'd prefer to have tightly focused tests, which look for specific problems. We know that we had a rendering bug that was causing plugin options to get omitted from usage strings - so this test tries to specifically check that that's not happening anymore. If it fails, we know exactly what the problem is. - Mark --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26688/#review56541 --- On Oct. 15, 2014, 12:41 p.m., Mark Chu-Carroll wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26688/ --- (Updated Oct. 15, 2014, 12:41 p.m.) Review request for Aurora, Joshua Cohen and Zameer Manji. Bugs: aurora-831 https://issues.apache.org/jira/browse/aurora-831 Repository: aurora Description --- - Put plugin-generated options into the correct order. - Include the option-name in the detailed help list. - Add missing metavars. Diffs - src/main/python/apache/aurora/client/cli/__init__.py da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8 src/main/python/apache/aurora/client/cli/options.py dc76c25b90acb9610e40b939e65c3cabf032649f src/main/python/apache/aurora/client/cli/standalone_client.py 20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1 src/test/python/apache/aurora/client/cli/test_help.py f73c8a3778b7d118ea2865f213b442a607fb4a7d Diff: https://reviews.apache.org/r/26688/diff/ Testing --- Thanks, Mark Chu-Carroll
Re: Review Request 26688: Fix errors in help rendering:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/#review56733
---
src/main/python/apache/aurora/client/cli/standalone_client.py
https://reviews.apache.org/r/26688/#comment97118
Is this a python standard for log levels? Could the help show all possible
values?
- Zameer Manji
On Oct. 15, 2014, 9:46 a.m., Mark Chu-Carroll wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/
---
(Updated Oct. 15, 2014, 9:46 a.m.)
Review request for Aurora, Joshua Cohen and Zameer Manji.
Bugs: aurora-831
https://issues.apache.org/jira/browse/aurora-831
Repository: aurora
Description
---
- Put plugin-generated options into the correct order.
- Include the option-name in the detailed help list.
- Add missing metavars.
Diffs
-
src/main/python/apache/aurora/client/cli/__init__.py
da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8
src/main/python/apache/aurora/client/cli/options.py
dc76c25b90acb9610e40b939e65c3cabf032649f
src/main/python/apache/aurora/client/cli/standalone_client.py
20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1
src/test/python/apache/aurora/client/cli/test_help.py
f73c8a3778b7d118ea2865f213b442a607fb4a7d
Diff: https://reviews.apache.org/r/26688/diff/
Testing
---
New help output:
{noformat}
[sun-wukong incubator-aurora (deprecate)]$ ./dist/aurora2.pex help cron
deschedule
Usage for verb cron deschedule:
deschedule [--verbose-logging] [--logging-level=numeric_level]
[--error-log-dir=error-log-dir] [--bind=var=value] CLUSTER/ROLE/ENV/NAME
Options:
--bind=var=value
Bind a pystachio variable name to a value. Multiple flags may be used
to specify multiple values.
CLUSTER/ROLE/ENV/NAME
Fully specified job key, in CLUSTER/ROLE/ENV/NAME format
--verbose-logging
Show verbose logging, including all logs up to level INFO (equivalent
to --logging-level=20)
--logging-level=numeric_level
Set logging to a specific numeric level.
--error-log-dir=error-log-dir
Directory location where error files containing stack traces should be
written. If the directory doesn't exist, it will be created
Remove the cron schedule for a job.
{noformat}
Thanks,
Mark Chu-Carroll
Re: Review Request 26688: Fix errors in help rendering:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/#review56737
---
src/main/python/apache/aurora/client/cli/standalone_client.py
https://reviews.apache.org/r/26688/#comment97119
It's the standard python loglevels. The number is basically any positive
integer from 0 to 50.
The underlying assumption here is that this is an option that's only going
to be used by someone who knows what they're doing, most likely someone
debugging the client. Users will generally use --verbose-logging instead of
setting a specific value.
Since this is an option that's going to show up in *every* command's help,
I'd really like to keep it's helpline as concise as possible.
How about adding numeric level as defined by python standard logging?
- Mark Chu-Carroll
On Oct. 15, 2014, 12:46 p.m., Mark Chu-Carroll wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/
---
(Updated Oct. 15, 2014, 12:46 p.m.)
Review request for Aurora, Joshua Cohen and Zameer Manji.
Bugs: aurora-831
https://issues.apache.org/jira/browse/aurora-831
Repository: aurora
Description
---
- Put plugin-generated options into the correct order.
- Include the option-name in the detailed help list.
- Add missing metavars.
Diffs
-
src/main/python/apache/aurora/client/cli/__init__.py
da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8
src/main/python/apache/aurora/client/cli/options.py
dc76c25b90acb9610e40b939e65c3cabf032649f
src/main/python/apache/aurora/client/cli/standalone_client.py
20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1
src/test/python/apache/aurora/client/cli/test_help.py
f73c8a3778b7d118ea2865f213b442a607fb4a7d
Diff: https://reviews.apache.org/r/26688/diff/
Testing
---
New help output:
{noformat}
[sun-wukong incubator-aurora (deprecate)]$ ./dist/aurora2.pex help cron
deschedule
Usage for verb cron deschedule:
deschedule [--verbose-logging] [--logging-level=numeric_level]
[--error-log-dir=error-log-dir] [--bind=var=value] CLUSTER/ROLE/ENV/NAME
Options:
--bind=var=value
Bind a pystachio variable name to a value. Multiple flags may be used
to specify multiple values.
CLUSTER/ROLE/ENV/NAME
Fully specified job key, in CLUSTER/ROLE/ENV/NAME format
--verbose-logging
Show verbose logging, including all logs up to level INFO (equivalent
to --logging-level=20)
--logging-level=numeric_level
Set logging to a specific numeric level.
--error-log-dir=error-log-dir
Directory location where error files containing stack traces should be
written. If the directory doesn't exist, it will be created
Remove the cron schedule for a job.
{noformat}
Thanks,
Mark Chu-Carroll
Re: Review Request 26716: Use JobKey when getting roles to authenticate against.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26716/#review56747 --- Ship it! Ship It! - Kevin Sweeney On Oct. 14, 2014, 5:23 p.m., Zameer Manji wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26716/ --- (Updated Oct. 14, 2014, 5:23 p.m.) Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-837 https://issues.apache.org/jira/browse/AURORA-837 Repository: aurora Description --- Use JobKey when getting roles to authenticate against. Diffs - src/main/java/org/apache/aurora/scheduler/base/Query.java d8572bb21a92025e7a51cf18d5bdf00fc1281078 src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java a43e5d7748c22d60f56f03a8a3d52949faebeff2 src/test/java/org/apache/aurora/scheduler/storage/mem/MemTaskStoreTest.java 3b61ff3c237de3ec5224a239d1756ce8d7093a52 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 0d51f7dc367081f72090736e36605bf363f3395e Diff: https://reviews.apache.org/r/26716/diff/ Testing --- ./gradlew clean test Thanks, Zameer Manji
Re: Review Request 26688: Fix errors in help rendering:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/
---
(Updated Oct. 15, 2014, 1:57 p.m.)
Review request for Aurora, Joshua Cohen and Zameer Manji.
Changes
---
Update description of the logging level parameter.
Bugs: aurora-831
https://issues.apache.org/jira/browse/aurora-831
Repository: aurora
Description
---
- Put plugin-generated options into the correct order.
- Include the option-name in the detailed help list.
- Add missing metavars.
Diffs (updated)
-
src/main/python/apache/aurora/client/cli/__init__.py
da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8
src/main/python/apache/aurora/client/cli/options.py
dc76c25b90acb9610e40b939e65c3cabf032649f
src/main/python/apache/aurora/client/cli/standalone_client.py
20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1
src/test/python/apache/aurora/client/cli/test_help.py
f73c8a3778b7d118ea2865f213b442a607fb4a7d
Diff: https://reviews.apache.org/r/26688/diff/
Testing
---
New help output:
{noformat}
[sun-wukong incubator-aurora (deprecate)]$ ./dist/aurora2.pex help cron
deschedule
Usage for verb cron deschedule:
deschedule [--verbose-logging] [--logging-level=numeric_level]
[--error-log-dir=error-log-dir] [--bind=var=value] CLUSTER/ROLE/ENV/NAME
Options:
--bind=var=value
Bind a pystachio variable name to a value. Multiple flags may be used
to specify multiple values.
CLUSTER/ROLE/ENV/NAME
Fully specified job key, in CLUSTER/ROLE/ENV/NAME format
--verbose-logging
Show verbose logging, including all logs up to level INFO (equivalent
to --logging-level=20)
--logging-level=numeric_level
Set logging to a specific numeric level.
--error-log-dir=error-log-dir
Directory location where error files containing stack traces should be
written. If the directory doesn't exist, it will be created
Remove the cron schedule for a job.
{noformat}
Thanks,
Mark Chu-Carroll
Re: Review Request 26688: Fix errors in help rendering:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/#review56750
---
Ship it!
Ship It!
- Zameer Manji
On Oct. 15, 2014, 10:57 a.m., Mark Chu-Carroll wrote:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/26688/
---
(Updated Oct. 15, 2014, 10:57 a.m.)
Review request for Aurora, Joshua Cohen and Zameer Manji.
Bugs: aurora-831
https://issues.apache.org/jira/browse/aurora-831
Repository: aurora
Description
---
- Put plugin-generated options into the correct order.
- Include the option-name in the detailed help list.
- Add missing metavars.
Diffs
-
src/main/python/apache/aurora/client/cli/__init__.py
da9d5b6ba4d22ba1f444341b97bbcfaf7889a4a8
src/main/python/apache/aurora/client/cli/options.py
dc76c25b90acb9610e40b939e65c3cabf032649f
src/main/python/apache/aurora/client/cli/standalone_client.py
20f4d7ef43ba336a2b6d02cbf5656c97bdfa2ea1
src/test/python/apache/aurora/client/cli/test_help.py
f73c8a3778b7d118ea2865f213b442a607fb4a7d
Diff: https://reviews.apache.org/r/26688/diff/
Testing
---
New help output:
{noformat}
[sun-wukong incubator-aurora (deprecate)]$ ./dist/aurora2.pex help cron
deschedule
Usage for verb cron deschedule:
deschedule [--verbose-logging] [--logging-level=numeric_level]
[--error-log-dir=error-log-dir] [--bind=var=value] CLUSTER/ROLE/ENV/NAME
Options:
--bind=var=value
Bind a pystachio variable name to a value. Multiple flags may be used
to specify multiple values.
CLUSTER/ROLE/ENV/NAME
Fully specified job key, in CLUSTER/ROLE/ENV/NAME format
--verbose-logging
Show verbose logging, including all logs up to level INFO (equivalent
to --logging-level=20)
--logging-level=numeric_level
Set logging to a specific numeric level.
--error-log-dir=error-log-dir
Directory location where error files containing stack traces should be
written. If the directory doesn't exist, it will be created
Remove the cron schedule for a job.
{noformat}
Thanks,
Mark Chu-Carroll
Re: Review Request 26458: Adding wait loop into host_drain status monitoring.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26458/#review56752 --- Ship it! src/main/python/apache/aurora/admin/host_maintenance.py https://reviews.apache.org/r/26458/#comment97138 minor nit: Amount(5, Time.MINUTES) - Brian Wickman On Oct. 10, 2014, 10 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26458/ --- (Updated Oct. 10, 2014, 10 p.m.) Review request for Aurora, Joe Smith and Brian Wickman. Bugs: AURORA-820 https://issues.apache.org/jira/browse/AURORA-820 Repository: aurora Description --- Throttling status check calls now at a predefined 5 second interval with a max timeout of 5 minutes. Diffs - src/main/python/apache/aurora/admin/host_maintenance.py 9c2a9f77109791da574e1624d27b6b7096a2678e src/test/python/apache/aurora/admin/test_host_maintenance.py 40228df59e43bc6034f2dc651c166a0c4b78aea8 src/test/python/apache/aurora/client/commands/test_maintenance.py d86aaf677804301fa5ddf1f76dba552f4fafb8c3 Diff: https://reviews.apache.org/r/26458/diff/ Testing --- ./pants src/test/python:all Thanks, Maxim Khutornenko
Re: Review Request 26753: Start removing clientv1.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26753/#review56756 --- Note: end-to-end tests are failing, but in the same way as they fail without this change. I'm working on debugging that, but it should be its own change/review, and I don't want to delay reviewing this. I agree. Probably goes without saying, but let's get the end-to-end tests green before putting this on master. - Bill Farner On Oct. 15, 2014, 4:20 p.m., Mark Chu-Carroll wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26753/ --- (Updated Oct. 15, 2014, 4:20 p.m.) Review request for Aurora, Bill Farner and Zameer Manji. Bugs: aurora-131 https://issues.apache.org/jira/browse/aurora-131 Repository: aurora Description --- - Make the aurora2 main target be the standalone clientv2; - Don't build the bridged client by default; but allow forced build using aurora2_bridge. - Modify cli tests so that they only depend on the standalone clientv2. - Modify vagrant config to make clientv2 the primary client. - Modify end-to-end tests to match the vagrant changes. (Note: end-to-end tests are failing, but in the same way as they fail without this change. I'm working on debugging that, but it should be its own change/review, and I don't want to delay reviewing this.) Diffs - docs/developing-aurora-client.md e1b2ccd7504f983169118a288721894184d67c97 examples/vagrant/aurorabuild.sh a27636655d722ca79f66b377fd847954d52e8feb examples/vagrant/provision-dev-cluster.sh 740bc212ba604b2c64af92eba1be41e8ed3fdbde src/main/python/apache/aurora/client/cli/BUILD 995570325bbb09ecbcc2ace5d223760c5d49367f src/test/python/apache/aurora/client/cli/test_api_from_cli.py a2b28ba23961284ba60358af54726e0386dd69b6 src/test/python/apache/aurora/client/cli/test_cancel_update.py e7052465411165acb3d5145664f2f166ac052500 src/test/python/apache/aurora/client/cli/test_command_hooks.py 9fc6fe2c2063cda494437d83044557b345acacea src/test/python/apache/aurora/client/cli/test_config_noun.py dfcbd7217b1d51609fa01c4d9cefed5471c91718 src/test/python/apache/aurora/client/cli/test_create.py 427f7ce4476b48d407b8bd2bf2c54c52e6e63079 src/test/python/apache/aurora/client/cli/test_cron.py c7b71c29d44150162fec8066947623fa91815424 src/test/python/apache/aurora/client/cli/test_diff.py 10817695352687cdb5b0c3ed9720e3091b230e68 src/test/python/apache/aurora/client/cli/test_help.py f73c8a3778b7d118ea2865f213b442a607fb4a7d src/test/python/apache/aurora/client/cli/test_inspect.py e997b9743b63d71f8624ecf5ca1dcae0227be70d src/test/python/apache/aurora/client/cli/test_kill.py bac4485fa105848d96e2505c4a2ea2eee45dc968 src/test/python/apache/aurora/client/cli/test_logging.py 9ca4dceeaa87d5fb2e38fe0d83fdcdf1ee597a0e src/test/python/apache/aurora/client/cli/test_open.py c20649f5cada241d0f6e9ae5f88d300eac073517 src/test/python/apache/aurora/client/cli/test_plugins.py dc5edd4f03cee062673231a04908193480c8071c src/test/python/apache/aurora/client/cli/test_quota.py 88fb9aec4d1eae6ad05da01752a670f902bafb1b src/test/python/apache/aurora/client/cli/test_restart.py a5f94484b30ecb8417116db9ce12c015957357c5 src/test/python/apache/aurora/client/cli/test_sla.py a1a3d8161ba747aa23a5e614e9ae31473d2058c1 src/test/python/apache/aurora/client/cli/test_status.py c704daec5a6eee73c7092a201b168881853908e8 src/test/python/apache/aurora/client/cli/test_supdate.py 6775c389cb1a0b80dd17fe179e8b98d4e9db0332 src/test/python/apache/aurora/client/cli/test_task_run.py 16fde14c03f6fd2c000e76625fad174835763f1b src/test/python/apache/aurora/client/cli/test_update.py cff1b6578aec6f5bcc1e610e58b47af233f32b41 src/test/sh/org/apache/aurora/e2e/test_common.sh 43d2516133c6d6cdb4236358f942396f057f739c src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh 324aa4dbeff00e673fe73b87e3a0766856cd213c src/test/sh/org/apache/aurora/e2e/test_end_to_end_v2.sh 8f4d2b01c9fa5a6ec9e8885a2d4fa0e9c3abb8a1 Diff: https://reviews.apache.org/r/26753/diff/ Testing --- - Ran all unit tests, confirmed they continue to pass. - Ran end-to-end tests (both v1 and v2 variants) with the updated setup, and verified that they fail in exactly the same way as before this change. Thanks, Mark Chu-Carroll
Review Request 26762: Deprecating Identity thrift struct (scheduler).
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26762/ --- Review request for Aurora, Kevin Sweeney and Bill Farner. Bugs: AURORA-84 https://issues.apache.org/jira/browse/AURORA-84 Repository: aurora Description --- Scheduler side of the Identity struct deprecation changes. Sending it as a separate CR for easier reviewing. Will have to be committed along with python changes (in-progress) to avoid breaking client diff functionality. Summary of the changes: * TaskConfig - dual write in StorageBackfill to populate new _key_ field. Incoming thrift objects are populated in ConfigurationManager during sanitizing. * TaskQuery - _owner_ to _role_ switch is handled in Query.Builder. All internal searching is now handled via _role_. * JobConfiguration - internal _owner_ refs redirected to _key.role_ 0.7.0. Diffs - src/main/java/org/apache/aurora/scheduler/MesosTaskFactory.java 83d0406a8bc7ccc1ae29804d2a4c8e8dfb90072c src/main/java/org/apache/aurora/scheduler/TaskIdGenerator.java 5c75cc8cae53edfa069c85c37ebad34774682081 src/main/java/org/apache/aurora/scheduler/TaskVars.java f1ab934541ad6d9ae74927f80a9c654a04922eb5 src/main/java/org/apache/aurora/scheduler/async/Preemptor.java e9f251508257cd7287ff00773e0073a3cd130df8 src/main/java/org/apache/aurora/scheduler/base/JobKeys.java a76c3fac71b35115064fba6644cff0066fd9e630 src/main/java/org/apache/aurora/scheduler/base/Query.java eded7a59eb394748b93d7fbc085a1bdf64b043cc src/main/java/org/apache/aurora/scheduler/base/Tasks.java 6ad79270c35c4fccb01f29d34ef1c4bbd7c953c8 src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java 865742171c11fbe5cf1469a69dd7258ec1be28c2 src/main/java/org/apache/aurora/scheduler/http/Utilization.java a0cb7bf56aeb7edd92b25d8d69a739d87452777a src/main/java/org/apache/aurora/scheduler/quota/QuotaManager.java 5f08997f04ffa7d9610c2b41551943b563412626 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 0f6731106c53420b92e60b9faf26c3614bd7ae00 src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java 58b94c2f2f3bac00f0692579974e8bdf159b6e40 src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 8c20ab6f2bebf1d1c0f91fed3f1e48361cdf45d6 src/main/java/org/apache/aurora/scheduler/storage/mem/MemTaskStore.java 37176237fac336413267f3c8bb4e1b9a6255150c src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 137f97d33decd14bf2f6dcdd9cd18c3db2b7c89c src/main/java/org/apache/aurora/scheduler/updater/JobUpdateControllerImpl.java 6ec130f4a9a5075b34452efb27c8fd0f08f93a63 src/main/thrift/org/apache/aurora/gen/api.thrift 8794731f4b3f1033588bdfa33c292e4796319a2a src/test/java/org/apache/aurora/scheduler/MesosTaskFactoryImplTest.java e96974764844b5d1a3a05f6996075fccee209594 src/test/java/org/apache/aurora/scheduler/TaskVarsTest.java 371ae87f5954fa5f092db1f6d21e2291d7576173 src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 606c4434b7158220ccf1403b6deac939021fee31 src/test/java/org/apache/aurora/scheduler/async/PreemptorImplTest.java 8ee84cda8670d117e2efa2d1a114da6f0d8315d6 src/test/java/org/apache/aurora/scheduler/async/TaskHistoryPrunerTest.java 53d2c6bb78ad08a84639c1ecd48ba64d17c3f9fc src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerImplTest.java 6534329a92bf005223fa8907cbe4a8a3a511e142 src/test/java/org/apache/aurora/scheduler/async/TaskSchedulerTest.java 9970948bace4c0ecbc51d6fc79270d77fb17bf87 src/test/java/org/apache/aurora/scheduler/cron/quartz/QuartzTestUtil.java d2d3e86bb5acf3402f55188b9ae440412ef14b5a src/test/java/org/apache/aurora/scheduler/quota/QuotaManagerImplTest.java 33790b118e788d7c894f7635f896619a3266192a src/test/java/org/apache/aurora/scheduler/sla/SlaTestUtil.java 21640f7ec2172d4c1b1bc744a4d71a6fa0a29376 src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java cdd29ea2b6fc92b967571028d299260556e16d42 src/test/java/org/apache/aurora/scheduler/stats/ResourceCounterTest.java fc12933cefdbfc03de5918de04172c0744d34588 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 80646a685ea918d80efafc5773e5805000a9c012 src/test/java/org/apache/aurora/scheduler/storage/mem/MemTaskStoreTest.java 3b61ff3c237de3ec5224a239d1756ce8d7093a52 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java c3f0bbe3745e24519438be7e9e73a4698061cb52 src/test/java/org/apache/aurora/scheduler/updater/JobUpdateEventSubscriberTest.java 5242a43b1c8a236b9420625a64ff24ad9ff75643 src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java f739e6d1b1af6eea4875e03d32bfe88cef87b3ff Diff: https://reviews.apache.org/r/26762/diff/ Testing --- ./gradlew -Pq build Thanks, Maxim
Re: Review Request 26764: Use python2.7 for thrift_wrapper_codegen.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26764/#review56767 --- Ship it! Thanks!! - Bill Farner On Oct. 15, 2014, 6:28 p.m., Brian Wickman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26764/ --- (Updated Oct. 15, 2014, 6:28 p.m.) Review request for Aurora and Bill Farner. Repository: aurora Description --- Use python2.7 for thrift_wrapper_codegen. Diffs - src/main/python/apache/aurora/tools/java/thrift_wrapper_codegen.py c0b109f30bd985a2822c117796cae10022ff180c Diff: https://reviews.apache.org/r/26764/diff/ Testing --- Thanks, Brian Wickman
Re: Review Request 26714: Remove use of the getVersion RPC from the client.
On Oct. 15, 2014, 1:13 a.m., Maxim Khutornenko wrote: Should we rather do it in 0.7.0 as deprecation warning suggests? If it's dropped now the client will be unable to talk to 0.5.0 scheduler. No thrift changes are made here. This is the client preparing for removal of the deprecated getVersion call. All fields used here are 0.5.0-compatible. - Bill --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/#review56633 --- On Oct. 14, 2014, 11:22 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/ --- (Updated Oct. 14, 2014, 11:22 p.m.) Review request for Aurora, Joshua Cohen and Maxim Khutornenko. Repository: aurora Description --- Remove use of the getVersion RPC from the client. Diffs - src/main/python/apache/aurora/client/api/scheduler_client.py 7f1c82bdbca427d1a09271b1e22f77f66da8e767 src/test/python/apache/aurora/client/api/test_restarter.py f1bf545a1aa1ab36f05fb0c6ea2ac7e4b1677932 src/test/python/apache/aurora/client/api/test_scheduler_client.py d78e7dca28d67997bc6c98cff619ab94a257c7dc src/test/python/apache/aurora/client/api/test_updater.py e8eaa9e6aa5fb3bc52a7195c26d9bd8294256780 src/test/python/apache/aurora/client/cli/test_api_from_cli.py a2b28ba23961284ba60358af54726e0386dd69b6 src/test/python/apache/aurora/client/fake_scheduler_proxy.py 12e70e9be9e3cf707f760ccd314c79825924c8bb Diff: https://reviews.apache.org/r/26714/diff/ Testing --- ./build-support/jenkins/build.sh Thanks, Bill Farner
Re: Review Request 26458: Adding wait loop into host_drain status monitoring.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26458/#review56774 --- src/test/python/apache/aurora/client/commands/test_maintenance.py https://reviews.apache.org/r/26458/#comment97177 mock_wait.assert_called_once_with - Joe Smith On Oct. 10, 2014, 3 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26458/ --- (Updated Oct. 10, 2014, 3 p.m.) Review request for Aurora, Joe Smith and Brian Wickman. Bugs: AURORA-820 https://issues.apache.org/jira/browse/AURORA-820 Repository: aurora Description --- Throttling status check calls now at a predefined 5 second interval with a max timeout of 5 minutes. Diffs - src/main/python/apache/aurora/admin/host_maintenance.py 9c2a9f77109791da574e1624d27b6b7096a2678e src/test/python/apache/aurora/admin/test_host_maintenance.py 40228df59e43bc6034f2dc651c166a0c4b78aea8 src/test/python/apache/aurora/client/commands/test_maintenance.py d86aaf677804301fa5ddf1f76dba552f4fafb8c3 Diff: https://reviews.apache.org/r/26458/diff/ Testing --- ./pants src/test/python:all Thanks, Maxim Khutornenko
Re: Review Request 26478: Add a flag to deduplicate storage snapshots
On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 56 https://reviews.apache.org/r/26478/diff/3/?file=721240#file721240line56 'reduplicate' doesn't sit well with me. Perhaps 'normalize' and 'denormalize' are more standard terms that apply? I don't feel too strongly, so don't change it if they seem equally good to you. I am decidedly ambivalent about the name On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 87 https://reviews.apache.org/r/26478/diff/3/?file=721240#file721240line87 This line is not covered in tests. Please address. However, i suggest you implement this as below, and inline. ScheduledTask partialScheduledTask = scheduledTask.deepCopy(); partialScheduledTask.getAssignedTask().unsetTaskConfig(); return partialScheduledTask; Inlined. On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 110 https://reviews.apache.org/r/26478/diff/2-3/?file=716380#file716380line110 Please use a better variable name. Inlined, so no variable to name. On Oct. 15, 2014, 11:54 a.m., Bill Farner wrote: docs/scheduler-storage.md, line 13 https://reviews.apache.org/r/26478/diff/3/?file=721234#file721234line13 Most users will want to enable both compression and deduplication. I suggest you yank this sentence out of this section, and add to the opening paragraph: The scheduler has two optimizations to reduce the size of snapshots and thus improve snapshot performance: compression and deduplication. Most users will want to enable both compression and deduplication. good idea, added - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/#review56762 --- On Oct. 14, 2014, 6:32 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/ --- (Updated Oct. 14, 2014, 6:32 p.m.) Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji. Bugs: AURORA-722 https://issues.apache.org/jira/browse/AURORA-722 Repository: aurora Description --- Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication ratio on Twitter's production snapshots. This format is backwards-incompatible, so this patch introduces a flag to control its use (defaulting off). This only changes the format used to write to the replicated log (where time is of the essence since all writes are done holding the global storage lock) - the format of backups written to disk is unchanged, as backups don't hold the lock. Diffs - config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 docs/scheduler-storage.md PRE-CREATION src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882 src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83 src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6 src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7 src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/26478/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 26714: Remove use of the getVersion RPC from the client.
On Oct. 15, 2014, 1:13 a.m., Maxim Khutornenko wrote: Should we rather do it in 0.7.0 as deprecation warning suggests? If it's dropped now the client will be unable to talk to 0.5.0 scheduler. Bill Farner wrote: No thrift changes are made here. This is the client preparing for removal of the deprecated getVersion call. All fields used here are 0.5.0-compatible. Never mind, I thought ServerInfo was added in 0.6.0. - Maxim --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/#review56633 --- On Oct. 15, 2014, 6:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/ --- (Updated Oct. 15, 2014, 6:48 p.m.) Review request for Aurora, Joshua Cohen and Maxim Khutornenko. Bugs: AURORA-142 https://issues.apache.org/jira/browse/AURORA-142 Repository: aurora Description --- Remove use of the getVersion RPC from the client. Diffs - src/main/python/apache/aurora/client/api/scheduler_client.py 7f1c82bdbca427d1a09271b1e22f77f66da8e767 src/test/python/apache/aurora/client/api/test_restarter.py f1bf545a1aa1ab36f05fb0c6ea2ac7e4b1677932 src/test/python/apache/aurora/client/api/test_scheduler_client.py d78e7dca28d67997bc6c98cff619ab94a257c7dc src/test/python/apache/aurora/client/api/test_updater.py e8eaa9e6aa5fb3bc52a7195c26d9bd8294256780 src/test/python/apache/aurora/client/cli/test_api_from_cli.py a2b28ba23961284ba60358af54726e0386dd69b6 src/test/python/apache/aurora/client/fake_scheduler_proxy.py 12e70e9be9e3cf707f760ccd314c79825924c8bb Diff: https://reviews.apache.org/r/26714/diff/ Testing --- ./build-support/jenkins/build.sh Thanks, Bill Farner
Re: Review Request 26714: Remove use of the getVersion RPC from the client.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/#review56786 --- Ship it! Ship It! - Maxim Khutornenko On Oct. 15, 2014, 6:48 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26714/ --- (Updated Oct. 15, 2014, 6:48 p.m.) Review request for Aurora, Joshua Cohen and Maxim Khutornenko. Bugs: AURORA-142 https://issues.apache.org/jira/browse/AURORA-142 Repository: aurora Description --- Remove use of the getVersion RPC from the client. Diffs - src/main/python/apache/aurora/client/api/scheduler_client.py 7f1c82bdbca427d1a09271b1e22f77f66da8e767 src/test/python/apache/aurora/client/api/test_restarter.py f1bf545a1aa1ab36f05fb0c6ea2ac7e4b1677932 src/test/python/apache/aurora/client/api/test_scheduler_client.py d78e7dca28d67997bc6c98cff619ab94a257c7dc src/test/python/apache/aurora/client/api/test_updater.py e8eaa9e6aa5fb3bc52a7195c26d9bd8294256780 src/test/python/apache/aurora/client/cli/test_api_from_cli.py a2b28ba23961284ba60358af54726e0386dd69b6 src/test/python/apache/aurora/client/fake_scheduler_proxy.py 12e70e9be9e3cf707f760ccd314c79825924c8bb Diff: https://reviews.apache.org/r/26714/diff/ Testing --- ./build-support/jenkins/build.sh Thanks, Bill Farner
Re: Review Request 26478: Add a flag to deduplicate storage snapshots
On Oct. 9, 2014, 10:30 a.m., Maxim Khutornenko wrote: src/main/thrift/org/apache/aurora/gen/storage.thrift, line 205 https://reviews.apache.org/r/26478/diff/2/?file=716383#file716383line205 Please, document fields. What is taskConfigId here? Documented all fields. On Oct. 9, 2014, 10:30 a.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 35 https://reviews.apache.org/r/26478/diff/2/?file=716380#file716380line35 This javadoc would highly benefit from some details about the source of duplication and a proposed solution. It's not obvious for a newcomer why TaskConfigs are duplicated. Added documentation elsewhere, happy to add more here if you think it's needed. On Oct. 9, 2014, 10:30 a.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 126 https://reviews.apache.org/r/26478/diff/2/?file=716380#file716380line126 Why result field here? refactored this code to be less performant, more readable On Oct. 9, 2014, 10:30 a.m., Maxim Khutornenko wrote: src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java, line 71 https://reviews.apache.org/r/26478/diff/2/?file=716386#file716386line71 How about a roundtrip test with no tasks in a snapshot? Good idea, added null-checking. On Oct. 9, 2014, 10:30 a.m., Maxim Khutornenko wrote: src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java, line 155 https://reviews.apache.org/r/26478/diff/2/?file=716380#file716380line155 Inverse log message of a hydration ratio would be useful here along with a Starting redupulication. Added. - Kevin --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/#review56010 --- On Oct. 15, 2014, 12:17 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/ --- (Updated Oct. 15, 2014, 12:17 p.m.) Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji. Bugs: AURORA-722 https://issues.apache.org/jira/browse/AURORA-722 Repository: aurora Description --- Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication ratio on Twitter's production snapshots. This format is backwards-incompatible, so this patch introduces a flag to control its use (defaulting off). This only changes the format used to write to the replicated log (where time is of the essence since all writes are done holding the global storage lock) - the format of backups written to disk is unchanged, as backups don't hold the lock. Diffs - config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 docs/scheduler-storage.md PRE-CREATION src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882 src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83 src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6 src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7 src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/26478/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 26478: Add a flag to deduplicate storage snapshots
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/#review56795 --- Ship it! Ship It! - Maxim Khutornenko On Oct. 15, 2014, 7:52 p.m., Kevin Sweeney wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/ --- (Updated Oct. 15, 2014, 7:52 p.m.) Review request for Aurora, David McLaughlin, Bill Farner, and Zameer Manji. Bugs: AURORA-722 https://issues.apache.org/jira/browse/AURORA-722 Repository: aurora Description --- Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication ratio on Twitter's production snapshots. This format is backwards-incompatible, so this patch introduces a flag to control its use (defaulting off). This only changes the format used to write to the replicated log (where time is of the essence since all writes are done holding the global storage lock) - the format of backups written to disk is unchanged, as backups don't hold the lock. Diffs - config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 docs/scheduler-storage.md PRE-CREATION src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882 src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83 src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6 src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7 src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/26478/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Re: Review Request 26664: Deprecating SANDBOX_DELETED task state.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26664/#review56813 --- Ship it! Ship It! - Joshua Cohen On Oct. 13, 2014, 11:22 p.m., Maxim Khutornenko wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26664/ --- (Updated Oct. 13, 2014, 11:22 p.m.) Review request for Aurora, Joshua Cohen and Bill Farner. Bugs: AURORA-751 https://issues.apache.org/jira/browse/AURORA-751 Repository: aurora Description --- Dropping the SANDBOX_DELETED from the scheduler. Diffs - src/main/java/org/apache/aurora/scheduler/async/GcExecutorLauncher.java cfab57896f9c76754ba3b42742504fb7e7a2cf79 src/main/java/org/apache/aurora/scheduler/base/Jobs.java 9ba83fa93409de7c6254bd8e7cc27e6bc10186e0 src/main/java/org/apache/aurora/scheduler/sla/SlaAlgorithm.java 0f6731106c53420b92e60b9faf26c3614bd7ae00 src/main/java/org/apache/aurora/scheduler/state/TaskStateMachine.java 86a8eb57ce7074f71d5212b34defe4320a5c430d src/main/java/org/apache/aurora/scheduler/storage/StorageBackfill.java 8c20ab6f2bebf1d1c0f91fed3f1e48361cdf45d6 src/main/python/apache/aurora/executor/gc_executor.py 9b40adaaa3634a451b4047915485e0e97d8f7914 src/main/resources/scheduler/assets/js/controllers.js 7e9037ee921b009dc2b7c5adcf057bedebb01632 src/main/resources/scheduler/assets/js/filters.js 7e8ca8408628d6f658da4267eb763e8fb4cb68c9 src/main/thrift/org/apache/aurora/gen/api.thrift 8794731f4b3f1033588bdfa33c292e4796319a2a src/test/java/org/apache/aurora/scheduler/app/SchedulerIT.java 606c4434b7158220ccf1403b6deac939021fee31 src/test/java/org/apache/aurora/scheduler/async/GcExecutorLauncherTest.java f2d153f446247032ad9d8d173fb70870dbfdcca1 src/test/java/org/apache/aurora/scheduler/async/TaskHistoryPrunerTest.java 53d2c6bb78ad08a84639c1ecd48ba64d17c3f9fc src/test/java/org/apache/aurora/scheduler/base/JobsTest.java f3b6ff0aa72c873d6b3cf3a1d327033c791ff6b9 src/test/java/org/apache/aurora/scheduler/sla/SlaAlgorithmTest.java eccf0c757d1e6addcd7619120f96ffa5f1ac38b5 src/test/java/org/apache/aurora/scheduler/state/StateManagerImplTest.java cdd29ea2b6fc92b967571028d299260556e16d42 src/test/java/org/apache/aurora/scheduler/state/TaskStateMachineTest.java cb3254728a697a963b92baff31b02dafa2be0039 src/test/java/org/apache/aurora/scheduler/storage/StorageBackfillTest.java 80646a685ea918d80efafc5773e5805000a9c012 Diff: https://reviews.apache.org/r/26664/diff/ Testing --- ./gradle -Pq build Verified UI in vagrant. Thanks, Maxim Khutornenko
Re: Review Request 26478: Add a flag to deduplicate storage snapshots
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/ --- (Updated Oct. 15, 2014, 2:32 p.m.) Review request for Aurora, Maxim Khutornenko, Bill Farner, and Zameer Manji. Changes --- -David, who is on vacation Bugs: AURORA-722 https://issues.apache.org/jira/browse/AURORA-722 Repository: aurora Description --- Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication ratio on Twitter's production snapshots. This format is backwards-incompatible, so this patch introduces a flag to control its use (defaulting off). This only changes the format used to write to the replicated log (where time is of the essence since all writes are done holding the global storage lock) - the format of backups written to disk is unchanged, as backups don't hold the lock. Diffs - config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 docs/scheduler-storage.md PRE-CREATION src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882 src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83 src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6 src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7 src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/26478/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
Review Request 26787: Re-add build message pointing to code coverage report.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26787/ --- Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- I accidentally dropped this message in https://reviews.apache.org/r/26787/ Diffs - build.gradle e436aaecb8540220f82d589f4858f2cad551fd78 Diff: https://reviews.apache.org/r/26787/diff/ Testing --- ./gradlew build, now points to the coverage report Thanks, Bill Farner
Re: Review Request 26787: Re-add build message pointing to code coverage report.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26787/#review56838 --- build.gradle https://reviews.apache.org/r/26787/#comment97272 Is it actually used anywhere? - Maxim Khutornenko On Oct. 15, 2014, 11:01 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26787/ --- (Updated Oct. 15, 2014, 11:01 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- I accidentally dropped this message in https://reviews.apache.org/r/26787/ Diffs - build.gradle e436aaecb8540220f82d589f4858f2cad551fd78 Diff: https://reviews.apache.org/r/26787/diff/ Testing --- ./gradlew build, now points to the coverage report Thanks, Bill Farner
Re: Review Request 26787: Re-add build message pointing to code coverage report.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26787/#review56840 --- Ship it! Ship It! - Kevin Sweeney On Oct. 15, 2014, 4:01 p.m., Bill Farner wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26787/ --- (Updated Oct. 15, 2014, 4:01 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- I accidentally dropped this message in https://reviews.apache.org/r/26787/ Diffs - build.gradle e436aaecb8540220f82d589f4858f2cad551fd78 Diff: https://reviews.apache.org/r/26787/diff/ Testing --- ./gradlew build, now points to the coverage report Thanks, Bill Farner
Re: Review Request 26478: Add a flag to deduplicate storage snapshots
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/26478/ --- (Updated Oct. 15, 2014, 5:05 p.m.) Review request for Aurora, Maxim Khutornenko, Bill Farner, and Zameer Manji. Changes --- Bill's feedback + fix markdown syntax Bugs: AURORA-722 https://issues.apache.org/jira/browse/AURORA-722 Repository: aurora Description --- Add a new format for deduplicated storage snapshots. Microbenchmarks show a 10x deduplication ratio on Twitter's production snapshots. This format is backwards-incompatible, so this patch introduces a flag to control its use (defaulting off). This only changes the format used to write to the replicated log (where time is of the essence since all writes are done holding the global storage lock) - the format of backups written to disk is unchanged, as backups don't hold the lock. Diffs (updated) - config/legacy_untested_classes.txt 3af99867eb25a7e44bb3520e82b1def125bd6e15 docs/deploying-aurora-scheduler.md 20f5f389f55e800cc5e6638e62e25cfb6e2d72b4 docs/scheduler-storage.md PRE-CREATION src/main/java/org/apache/aurora/codec/ThriftBinaryCodec.java 65e986eaa2c4193431ca048425a1ed3ab60f5882 src/main/java/org/apache/aurora/scheduler/storage/log/EntrySerializer.java 7239a6a5eb5479e395e16423c83fdf80a77e5a83 src/main/java/org/apache/aurora/scheduler/storage/log/LogManager.java 4b50e2069407dc263b4fc93f1827d3a8836253bf src/main/java/org/apache/aurora/scheduler/storage/log/LogStorage.java f806297d1d0700155c976743f936b2b8a3a390fb src/main/java/org/apache/aurora/scheduler/storage/log/LogStorageModule.java 769348e6b8a5c701734afff391b1c77de35222c6 src/main/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicator.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/storage/log/StreamManager.java 22db80eaf34fe736fa5a3a9289836c9ac9e59906 src/main/java/org/apache/aurora/scheduler/storage/log/StreamManagerImpl.java e5cfbf5cf43bf5bbc38c42fe685a7e9f0d03af2a src/main/thrift/org/apache/aurora/gen/storage.thrift 5350ec945fbe028ee4641683815a068ce00b5efc src/test/java/org/apache/aurora/scheduler/storage/log/LogManagerTest.java 39729b374fe4e383f9b5ada7d016923766df9af7 src/test/java/org/apache/aurora/scheduler/storage/log/LogStorageTest.java 7a8c3b882633376a1bf6a78616d55aaa7401d13f src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotDeduplicatorImplTest.java PRE-CREATION Diff: https://reviews.apache.org/r/26478/diff/ Testing --- ./gradlew -Pq build Thanks, Kevin Sweeney
