Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/ --- (Updated March 24, 2017, 4:38 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji. Repository: aurora Description --- This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command. If there is consensus, this can be added to all commands as well. Diffs (updated) - src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef Diff: https://reviews.apache.org/r/54957/diff/5/ Changes: https://reviews.apache.org/r/54957/diff/4-5/ Testing --- * Manual testing * ./build-support/jenkins/build.sh passes Thanks, Karthik Anantha Padmanabhan
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
> On March 23, 2017, 4:52 p.m., David McLaughlin wrote: > > src/main/python/apache/aurora/admin/admin.py > > Lines 244 (patched) > > <https://reviews.apache.org/r/54957/diff/4/?file=1672873#file1672873line244> > > > > Where is options.no_retry defined? duh ! Bad push my bad. - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/#review169894 --- On March 23, 2017, 4:23 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54957/ > --- > > (Updated March 23, 2017, 4:23 p.m.) > > > Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar > Shanmugham, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > This diff adds an option to not retry api calls to the scheduler. For some of > the non-idempotent operations we would like to not automatically retry. This > patch makes this functionality available only to the `schedule_backup_now` > command. > > If there is consensus, this can be added to all commands as well. > > > Diffs > - > > src/main/python/apache/aurora/admin/admin.py > 070c348d2ca5db1edecf832efd9aa5481bddaa4b > src/main/python/apache/aurora/client/api/__init__.py > e1dde638bd1d686269fbcd88cb083a52e7f5dbfc > src/main/python/apache/aurora/client/api/scheduler_client.py > 9bbfece012e48e0b1752bbefd25c89e04d312cf6 > src/test/python/apache/aurora/client/api/test_scheduler_client.py > f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef > > > Diff: https://reviews.apache.org/r/54957/diff/4/ > > > Testing > --- > > * Manual testing > * ./build-support/jenkins/build.sh passes > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/ --- (Updated March 23, 2017, 4:23 p.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji. Changes --- Style changes. Repository: aurora Description --- This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command. If there is consensus, this can be added to all commands as well. Diffs (updated) - src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef Diff: https://reviews.apache.org/r/54957/diff/4/ Changes: https://reviews.apache.org/r/54957/diff/3-4/ Testing --- * Manual testing * ./build-support/jenkins/build.sh passes Thanks, Karthik Anantha Padmanabhan
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/ --- (Updated March 23, 2017, 8:23 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji. Repository: aurora Description --- This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command. If there is consensus, this can be added to all commands as well. Diffs - src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef Diff: https://reviews.apache.org/r/54957/diff/3/ Testing (updated) --- * Manual testing * ./build-support/jenkins/build.sh passes Thanks, Karthik Anantha Padmanabhan
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/ --- (Updated March 23, 2017, 8:23 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji. Changes --- Rename tests. Repository: aurora Description --- This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command. If there is consensus, this can be added to all commands as well. Diffs (updated) - src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef Diff: https://reviews.apache.org/r/54957/diff/3/ Changes: https://reviews.apache.org/r/54957/diff/2-3/ Testing --- * Manuall testing * ./build-support/jenkins/build.sh passes Thanks, Karthik Anantha Padmanabhan
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/ --- (Updated March 23, 2017, 8:22 a.m.) Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar Shanmugham, and Zameer Manji. Changes --- CR Comments. Repository: aurora Description --- This diff adds an option to not retry api calls to the scheduler. For some of the non-idempotent operations we would like to not automatically retry. This patch makes this functionality available only to the `schedule_backup_now` command. If there is consensus, this can be added to all commands as well. Diffs (updated) - src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 src/main/python/apache/aurora/client/api/__init__.py e1dde638bd1d686269fbcd88cb083a52e7f5dbfc src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef Diff: https://reviews.apache.org/r/54957/diff/2/ Changes: https://reviews.apache.org/r/54957/diff/1-2/ Testing --- * Manuall testing * ./build-support/jenkins/build.sh passes Thanks, Karthik Anantha Padmanabhan
Re: Review Request 55237: WIP for idempotent api endpoints.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/55237/ --- (Updated Jan. 10, 2017, 11:21 p.m.) Review request for Aurora and Santhosh Kumar Shanmugham. Repository: aurora Description --- WIP for idempotent requests. The changes are basically the following: 1. In the client we construct a thrift transport based on the requests library. A new `TRequestsTransportRetry` adds a http header called `Idempotency-Token`. The retry logic is moved here so that retries will use the same idempotency token. I use the retry library for exponential backoff 2. In the scheduler `TContentAwareServlet` is provided with a `RequestCache` which is nothing but a Guava cache which expires keys after an hour. Diffs - 3rdparty/python/requirements.txt df2db7ac1a1d62465731c057f67000cb589a RELEASE-NOTES.md 90c4793b218f54ad92ab4da5049d5e92c3a104b3 src/main/java/org/apache/aurora/scheduler/http/api/ApiModule.java e46820952fb6028911bca924169ceade6a134bfc src/main/java/org/apache/aurora/scheduler/http/api/RequestCache.java PRE-CREATION src/main/java/org/apache/aurora/scheduler/http/api/TContentAwareServlet.java d9df96896d83786596880105968dc7fcdc0168e5 src/main/python/apache/aurora/admin/BUILD b5d37f718a25d43d7ac07e30f789cf526bdfdcc2 src/main/python/apache/aurora/admin/admin.py 070c348d2ca5db1edecf832efd9aa5481bddaa4b src/main/python/apache/aurora/admin/aurora_admin.py fbebbab8c827b5695042d18770d850e31fc38122 src/main/python/apache/aurora/client/api/scheduler_client.py 9bbfece012e48e0b1752bbefd25c89e04d312cf6 src/main/python/apache/aurora/common/BUILD 0e4c51020946dc21953493bd43b944177c444c28 src/main/python/apache/aurora/common/transport.py f44d13afa06112ff7b03a68cd01e7d9ef9e550ea src/test/java/org/apache/aurora/scheduler/http/api/TContentAwareServletTests.java PRE-CREATION src/test/python/apache/aurora/client/api/test_scheduler_client.py f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef src/test/python/apache/aurora/common/test_transport.py c3d35d278a68db603954ad1e3cf26c6359ae3926 Diff: https://reviews.apache.org/r/55237/diff/ Testing --- Works locally. Not good for a review , this is just work in progress. Thanks, Karthik Anantha Padmanabhan
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
> On Jan. 4, 2017, 11:54 p.m., Karthik Anantha Padmanabhan wrote: > > I was going to wrap this up - but how do people feel about making the all > > endpoints "idempotent" by the following method ? Inlcude an > > "idempotency-token" along as part of the HTTP header. This token is locally > > cached for, say, an hour. Every request with the same token will return > > simply short circuit and not be processed. The retry logic to transport > > layer so that we can transparently add the idempotency tokens. > > Santhosh Kumar Shanmugham wrote: > Can you explain this in detail? Particularly flesh out the `HTTP headers` > (aren't we using a Thrift interface while speaking to the Scheduler?) and the > `locally cached` (have the Scheduler cache the `idempotency-token` or on the > client?) parts. > > I like this approach better, which can fix the issue across all APIs. https://reviews.apache.org/r/55237/ is a WIP for idempotent requests. Yes we still use HTTP as the transport but just stuff the thrift binary in the body of a POST request. - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/#review160546 --------------- On Dec. 22, 2016, 12:06 a.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54957/ > --- > > (Updated Dec. 22, 2016, 12:06 a.m.) > > > Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar > Shanmugham, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > This diff adds an option to not retry api calls to the scheduler. For some of > the non-idempotent operations we would like to not automatically retry. This > patch makes this functionality available only to the `schedule_backup_now` > command. > > If there is consensus, this can be added to all commands as well. > > > Diffs > - > > src/main/python/apache/aurora/admin/admin.py > 070c348d2ca5db1edecf832efd9aa5481bddaa4b > src/main/python/apache/aurora/admin/aurora_admin.py > fbebbab8c827b5695042d18770d850e31fc38122 > src/main/python/apache/aurora/client/api/__init__.py > e1dde638bd1d686269fbcd88cb083a52e7f5dbfc > src/main/python/apache/aurora/client/api/scheduler_client.py > 9bbfece012e48e0b1752bbefd25c89e04d312cf6 > src/test/python/apache/aurora/client/api/test_scheduler_client.py > f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef > > Diff: https://reviews.apache.org/r/54957/diff/ > > > Testing > --- > > * Manuall testing > * ./build-support/jenkins/build.sh passes > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
> On Dec. 22, 2016, 12:54 a.m., Karthik Anantha Padmanabhan wrote: > > src/main/python/apache/aurora/admin/aurora_admin.py, line 55 > > <https://reviews.apache.org/r/54957/diff/1/?file=1590757#file1590757line55> > > > > In the current implementation we retry only when a `TimeoutError` is > > raised. > > > > And we raise `TimeoutError` when trying to construct the client and > > connecting to the leader/ZK. And this might involve things like unable to > > resolve the leader url etc. So when an API call timesout we do not retry. - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/#review159896 --------------- On Dec. 22, 2016, 12:06 a.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54957/ > --- > > (Updated Dec. 22, 2016, 12:06 a.m.) > > > Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar > Shanmugham, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > This diff adds an option to not retry api calls to the scheduler. For some of > the non-idempotent operations we would like to not automatically retry. This > patch makes this functionality available only to the `schedule_backup_now` > command. > > If there is consensus, this can be added to all commands as well. > > > Diffs > - > > src/main/python/apache/aurora/admin/admin.py > 070c348d2ca5db1edecf832efd9aa5481bddaa4b > src/main/python/apache/aurora/admin/aurora_admin.py > fbebbab8c827b5695042d18770d850e31fc38122 > src/main/python/apache/aurora/client/api/__init__.py > e1dde638bd1d686269fbcd88cb083a52e7f5dbfc > src/main/python/apache/aurora/client/api/scheduler_client.py > 9bbfece012e48e0b1752bbefd25c89e04d312cf6 > src/test/python/apache/aurora/client/api/test_scheduler_client.py > f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef > > Diff: https://reviews.apache.org/r/54957/diff/ > > > Testing > --- > > * Manuall testing > * ./build-support/jenkins/build.sh passes > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 54957: Add option to not retry api calls to the scheduler.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54957/#review159896 --- src/main/python/apache/aurora/admin/aurora_admin.py (line 55) <https://reviews.apache.org/r/54957/#comment230947> In the current implementation we retry only when a `TimeoutError` is raised. And we raise `TimeoutError` when trying to construct the client and connecting to the leader/ZK. And this might involve things like unable to resolve the leader url etc. - Karthik Anantha Padmanabhan On Dec. 22, 2016, 12:06 a.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54957/ > --- > > (Updated Dec. 22, 2016, 12:06 a.m.) > > > Review request for Aurora, David McLaughlin, Joshua Cohen, Santhosh Kumar > Shanmugham, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > This diff adds an option to not retry api calls to the scheduler. For some of > the non-idempotent operations we would like to not automatically retry. This > patch makes this functionality available only to the `schedule_backup_now` > command. > > If there is consensus, this can be added to all commands as well. > > > Diffs > - > > src/main/python/apache/aurora/admin/admin.py > 070c348d2ca5db1edecf832efd9aa5481bddaa4b > src/main/python/apache/aurora/admin/aurora_admin.py > fbebbab8c827b5695042d18770d850e31fc38122 > src/main/python/apache/aurora/client/api/__init__.py > e1dde638bd1d686269fbcd88cb083a52e7f5dbfc > src/main/python/apache/aurora/client/api/scheduler_client.py > 9bbfece012e48e0b1752bbefd25c89e04d312cf6 > src/test/python/apache/aurora/client/api/test_scheduler_client.py > f6018caa4f431e85a9e9ff203ac3d4b6c33f40ef > > Diff: https://reviews.apache.org/r/54957/diff/ > > > Testing > --- > > * Manuall testing > * ./build-support/jenkins/build.sh passes > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 54288: Make leader elections resilient to ZK disconnections.
> On Dec. 2, 2016, 10:37 p.m., Zameer Manji wrote: > > Can someone else validate that the tests pass for them locally? I can't > > reproduce the jenkins failure. I wonder if this is a timing issue in the tests ? `expectGroupEvent(PathChildrenCacheEvent.Type.CHILD_ADDED)` returns as soon as there is a CHILD_ADDED event. Is it possible `listener.onLeading` isn't called by then ? assertTrue(capture.hasCaptured()) - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/54288/#review157847 --- On Dec. 2, 2016, 3:19 a.m., Zameer Manji wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/54288/ > --- > > (Updated Dec. 2, 2016, 3:19 a.m.) > > > Review request for Aurora, David McLaughlin, Joshua Cohen, John Sirois, and > Stephan Erb. > > > Bugs: AURORA-1669 > https://issues.apache.org/jira/browse/AURORA-1669 > > > Repository: aurora > > > Description > --- > > As documented in AURORA-1840 the Curator `LeaderLatch` recipe abdicates > leadership if the ZK connection is lost or if there is a timeout. This is not > compatible with the commons based implementation which would only abdicate > leadership if the ZK session timeout occurred. > > This replaces the `LeaderLatch` recipe with the `LeaderSelector` recipe with a > custom listener that only loses leadership if a connection loss occurs. > > > Diffs > - > > > commons/src/main/java/org/apache/aurora/common/zookeeper/testing/ZooKeeperTestServer.java > 50acaeba82e163f8f2970a264cbd889c9eb3b5ed > > src/main/java/org/apache/aurora/scheduler/discovery/CuratorSingletonService.java > c378172c850aafe0a9381552b5067277b40dbfab > > src/test/java/org/apache/aurora/scheduler/discovery/BaseCuratorDiscoveryTest.java > a2b4125369d1f6c0a79bc4ac0fb3d2dab8a6c583 > > src/test/java/org/apache/aurora/scheduler/discovery/CuratorSingletonServiceTest.java > 6ea49b0c690d288ff59d1d4798144bfa2d153d3a > > Diff: https://reviews.apache.org/r/54288/diff/ > > > Testing > --- > > > Thanks, > > Zameer Manji > >
Review Request 51900: Add reconciliation_explicit_batch_size scheduler option to the documentation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51900/ --- Review request for Aurora and Joshua Cohen. Repository: aurora Description --- The reconciliation_explicit_batch_size was missing from the scheduler coniguration documentation. Given that this option is referenced in the `aurora_admin` command `reconcile_tasks` it will be good to have it. Diffs - docs/reference/scheduler-configuration.md 31be71491de00e53ce4764e480ff2728d06a3f1d Diff: https://reviews.apache.org/r/51900/diff/ Testing --- ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 12, 2016, 9:29 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- * Use Precondition to check validity of settings argument * Make reconciliation type checking more explicit Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 12, 2016, 5:49 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 9, 2016, 6:43 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 9, 2016, 6:32 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- * Address review comments from Zameer, Maxim. * Add two new UTs. * Validate the batch size input. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 8, 2016, 6:43 p.m., Santhosh Kumar Shanmugham wrote: > > src/test/python/apache/aurora/admin/test_admin.py, line 285 > > <https://reviews.apache.org/r/51662/diff/7/?file=1494557#file1494557line285> > > > > Should this also be assert_called_once_with(None) ? No - I set the option as 500 in the test and I want to test that we call with the provided option - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148230 --- On Sept. 8, 2016, 5:27 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 8, 2016, 5:27 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 8, 2016, 6:18 p.m., Maxim Khutornenko wrote: > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java, > > line 123 > > <https://reviews.apache.org/r/51662/diff/7/?file=1494551#file1494551line123> > > > > Prefer using immutable `IExplicit...` wrapper generated for you by the > > build. With the `IExplicit...` version I do not have the ability to check if the field is set or not using `isSet...`. I use that in order to provide the default values for the batch size. > On Sept. 8, 2016, 6:18 p.m., Maxim Khutornenko wrote: > > src/main/python/apache/aurora/admin/admin.py, line 351 > > <https://reviews.apache.org/r/51662/diff/7/?file=1494553#file1494553line351> > > > > Needs clarifcation that `reconciliation_explicit_batch_size` is a > > scheduler setting. Good question. It does not show up in the docs. However it is a command line argument in the `ReconciliationModule`. So should this be a scheduler setting ? - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148225 --------------- On Sept. 8, 2016, 5:27 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 8, 2016, 5:27 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > RELEASE-NOTES.md bbf7198dc7ce53bb02a1bc59f75a661e23472913 > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 8, 2016, 2:11 a.m., Joshua Cohen wrote: > > src/main/python/apache/aurora/admin/admin.py, lines 357-358 > > <https://reviews.apache.org/r/51662/diff/5/?file=1493867#file1493867line357> > > > > What if type is `foo`, etc? We should ensure the value is valid rather > > than defaulting to explicit reconciliation. Santhosh's suggestion should fix this ? - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148128 --- On Sept. 8, 2016, 6:48 a.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 8, 2016, 6:48 a.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 8, 2016, 6:48 a.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- * Address code review style comments * Add an entry in README.md Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - RELEASE-NOTES.md d79aaadc197697d09a71c83494a01765d6a983d4 api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 8, 2016, 12:30 a.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- * Introduce the struct for Explicit Reconcile settings. * Update the client batch size defaults to None. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 7, 2016, 9:45 p.m., Santhosh Kumar Shanmugham wrote: > > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 1209-1214 > > <https://reviews.apache.org/r/51662/diff/4/?file=1493667#file1493667line1209> > > > > Did we consider combining the "explicit" and "implicit" reconciliation > > RPCs into a single "reconcile" RPC with an optional batchSize argument, so > > that it maps to the mesos master API. > > > > http://mesos.apache.org/documentation/latest/reconciliation/ > > Karthik Anantha Padmanabhan wrote: > We have more than one option on the scheduler for the explicit reconcile. > If at all we want to expose those to the admin CLI in the future it may be > better to have them as two separate API's. > > Santhosh Kumar Shanmugham wrote: > "We have more than one option on the scheduler for the explicit > reconcile." - can you given an example of what you mean by this. > > Karthik Anantha Padmanabhan wrote: > There is another releavant parameter called `explicitBatchDelaySeconds` > which dictates how long we wait in between batches before we call reconcile > on mesos. > > But it is possible that we may add another option in the future ? In that > case wouldn't it be better to have two separate API's ? Hmm actually if we are passing in a struct of `ExplicitReconcileSettings` , then adding options will be easier. So I can check `isNull` on the settings object on the scheduler to identify an implicit reconcile. I can go either way. I just don't know if there is a strong reason to make it a single API as opposed to two. - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148089 --- On Sept. 7, 2016, 9:08 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 7, 2016, 9:08 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 7, 2016, 9:45 p.m., Santhosh Kumar Shanmugham wrote: > > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 1209-1214 > > <https://reviews.apache.org/r/51662/diff/4/?file=1493667#file1493667line1209> > > > > Did we consider combining the "explicit" and "implicit" reconciliation > > RPCs into a single "reconcile" RPC with an optional batchSize argument, so > > that it maps to the mesos master API. > > > > http://mesos.apache.org/documentation/latest/reconciliation/ > > Karthik Anantha Padmanabhan wrote: > We have more than one option on the scheduler for the explicit reconcile. > If at all we want to expose those to the admin CLI in the future it may be > better to have them as two separate API's. > > Santhosh Kumar Shanmugham wrote: > "We have more than one option on the scheduler for the explicit > reconcile." - can you given an example of what you mean by this. There is another releavant parameter called `explicitBatchDelaySeconds` which dictates how long we wait in between batches before we call reconcile on mesos. But it is possible that we may add another option in the future ? In that case wouldn't it be better to have two separate API's ? - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148089 --- On Sept. 7, 2016, 9:08 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 7, 2016, 9:08 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 7, 2016, 9:45 p.m., Santhosh Kumar Shanmugham wrote:
> > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 1209-1214
> > <https://reviews.apache.org/r/51662/diff/4/?file=1493667#file1493667line1209>
> >
> > Did we consider combining the "explicit" and "implicit" reconciliation
> > RPCs into a single "reconcile" RPC with an optional batchSize argument, so
> > that it maps to the mesos master API.
> >
> > http://mesos.apache.org/documentation/latest/reconciliation/
We have more than one option on the scheduler for the explicit reconcile. If at
all we want to expose those to the admin CLI in the future it may be better to
have them as two separate API's.
> On Sept. 7, 2016, 9:45 p.m., Santhosh Kumar Shanmugham wrote:
> > src/main/python/apache/aurora/admin/admin.py, line 342
> > <https://reviews.apache.org/r/51662/diff/4/?file=1493670#file1493670line342>
> >
> > 'default=0' here and the usage doc reads default as 1000.
We want the default value to match the one from settings.explicitBatchSize on
the scheduler. So on the scheduler we use the default value if the value here
is 0. Although I agree that the 0 here is a bit misleading.
I like the optional batchSize argument for the explicit reconcile RPC.
I am thinking there can be a
struct ExplicitReconcileSettings {
1. optional i32 batchSize;
}
This can let us add more explicit reconcile settings in the future.
Also does thrift lets us declare optional parameters in a function (unless it
is wrapped in a struct) ?
- Karthik
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/51662/#review148089
---
On Sept. 7, 2016, 9:08 p.m., Karthik Anantha Padmanabhan wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/51662/
> ---
>
> (Updated Sept. 7, 2016, 9:08 p.m.)
>
>
> Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> AURORA-1602 Aurora admin commands for reconcilation
>
>
> Diffs
> -
>
> api/src/main/thrift/org/apache/aurora/gen/api.thrift
> c5765b70501c101f0535b4eed94e9948c36808f9
>
> src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java
> 3275d72a0a74909e635bce615e2036ec72aa38ee
>
> src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java
> 929d021a336c6a3438613c9340c84a1096dc9069
> src/main/python/apache/aurora/admin/admin.py
> 76009b9c1c7a130c25abad544a176dc590dafb12
> src/main/python/apache/aurora/client/api/__init__.py
> 9149c3018ae58d405f284fcbd4076d251ccc8192
>
> src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java
> b9317dc20456f90057ec2bf4d10619a5ae986187
>
> src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java
> 779dc302602ae8842084807ca868a491ea99b676
> src/test/python/apache/aurora/admin/test_admin.py
> eb193c4a33f5275f05a995338446bec0e19bc1cf
> src/test/python/apache/aurora/client/api/test_scheduler_client.py
> afac2500551af2fce406bb906aa4e33f353e90a1
>
> Diff: https://reviews.apache.org/r/51662/diff/
>
>
> Testing
> ---
>
> * Manually tested on my local vagrant installation
> * ./build-support/jenkins/build.sh
>
>
> Thanks,
>
> Karthik Anantha Padmanabhan
>
>
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 7, 2016, 9:08 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 7, 2016, 9:01 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- Address code review comments Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
> On Sept. 7, 2016, 6:14 p.m., Maxim Khutornenko wrote: > > api/src/main/thrift/org/apache/aurora/gen/api.thrift, lines 944-950 > > <https://reviews.apache.org/r/51662/diff/2/?file=1492509#file1492509line944> > > > > I don't think there is much value in having this struct and the related > > `reconcileStatus` RPC. Cluster operators are normally relying on scheduler > > stats to monitor activities like this. Cool makes sense > On Sept. 7, 2016, 6:14 p.m., Maxim Khutornenko wrote: > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java, > > lines 135-143 > > <https://reviews.apache.org/r/51662/diff/2/?file=1492510#file1492510line135> > > > > This only prevents overlapping explicit or implicit runs but does not > > prevent explicit and implicit overlapping runs. Also, this does not cancel > > any calls already waiting for a lock. > > > > Given that we deal with admin-triggered events (who are expected to > > know what they do), I suggest we simplify the logic here by simply > > scheduling an out of band run and not try to synchronize anything around > > reconciliation runs. This would let you cut the bulk of the changes below. okay agree. > On Sept. 7, 2016, 6:14 p.m., Maxim Khutornenko wrote: > > src/main/python/apache/aurora/admin/admin.py, line 355 > > <https://reviews.apache.org/r/51662/diff/2/?file=1492512#file1492512line355> > > > > The default here would be 0 if you accept my suggestion above. Okay I am assuming that an operator will never need to explicitly run reconcilation with a batch size of 0. Else I could choose -1 ? - Karthik --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review148046 --- On Sept. 6, 2016, 11:34 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 6, 2016, 11:34 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/admin/admin_util.py > 394deb57af9ad8832a02ceab15f33b3c1e5c902b > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/#review147953 --- @ReviewBot retry - Karthik Anantha Padmanabhan On Sept. 6, 2016, 11:34 p.m., Karthik Anantha Padmanabhan wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51662/ > --- > > (Updated Sept. 6, 2016, 11:34 p.m.) > > > Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > AURORA-1602 Aurora admin commands for reconcilation > > > Diffs > - > > api/src/main/thrift/org/apache/aurora/gen/api.thrift > c5765b70501c101f0535b4eed94e9948c36808f9 > > src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java > 3275d72a0a74909e635bce615e2036ec72aa38ee > > src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java > 929d021a336c6a3438613c9340c84a1096dc9069 > src/main/python/apache/aurora/admin/admin.py > 76009b9c1c7a130c25abad544a176dc590dafb12 > src/main/python/apache/aurora/admin/admin_util.py > 394deb57af9ad8832a02ceab15f33b3c1e5c902b > src/main/python/apache/aurora/client/api/__init__.py > 9149c3018ae58d405f284fcbd4076d251ccc8192 > > src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java > b9317dc20456f90057ec2bf4d10619a5ae986187 > > src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java > 779dc302602ae8842084807ca868a491ea99b676 > src/test/python/apache/aurora/admin/test_admin.py > eb193c4a33f5275f05a995338446bec0e19bc1cf > src/test/python/apache/aurora/client/api/test_scheduler_client.py > afac2500551af2fce406bb906aa4e33f353e90a1 > > Diff: https://reviews.apache.org/r/51662/diff/ > > > Testing > --- > > * Manually tested on my local vagrant installation > * ./build-support/jenkins/build.sh > > > Thanks, > > Karthik Anantha Padmanabhan > >
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 6, 2016, 11:34 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Changes --- Updated Testing done. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/admin/admin_util.py 394deb57af9ad8832a02ceab15f33b3c1e5c902b src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing (updated) --- * Manually tested on my local vagrant installation * ./build-support/jenkins/build.sh Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 6, 2016, 11:18 p.m.) Review request for Aurora, Joshua Cohen, Maxim Khutornenko, and Zameer Manji. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs (updated) - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/admin/admin_util.py 394deb57af9ad8832a02ceab15f33b3c1e5c902b src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 src/test/python/apache/aurora/admin/test_admin.py eb193c4a33f5275f05a995338446bec0e19bc1cf src/test/python/apache/aurora/client/api/test_scheduler_client.py afac2500551af2fce406bb906aa4e33f353e90a1 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- * Manually tested on my local vagrant installation Thanks, Karthik Anantha Padmanabhan
Re: Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- (Updated Sept. 6, 2016, 6:42 p.m.) Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/admin/admin_util.py 394deb57af9ad8832a02ceab15f33b3c1e5c902b src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 Diff: https://reviews.apache.org/r/51662/diff/ Testing (updated) --- * Manually tested on my local vagrant installation Thanks, Karthik Anantha Padmanabhan
Review Request 51662: AURORA-1602 Aurora admin commands for reconcilation
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51662/ --- Review request for Aurora and Maxim Khutornenko. Repository: aurora Description --- AURORA-1602 Aurora admin commands for reconcilation Diffs - api/src/main/thrift/org/apache/aurora/gen/api.thrift c5765b70501c101f0535b4eed94e9948c36808f9 src/main/java/org/apache/aurora/scheduler/reconciliation/TaskReconciler.java 3275d72a0a74909e635bce615e2036ec72aa38ee src/main/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterface.java 929d021a336c6a3438613c9340c84a1096dc9069 src/main/python/apache/aurora/admin/admin.py 76009b9c1c7a130c25abad544a176dc590dafb12 src/main/python/apache/aurora/admin/admin_util.py 394deb57af9ad8832a02ceab15f33b3c1e5c902b src/main/python/apache/aurora/client/api/__init__.py 9149c3018ae58d405f284fcbd4076d251ccc8192 src/test/java/org/apache/aurora/scheduler/reconciliation/TaskReconcilerTest.java b9317dc20456f90057ec2bf4d10619a5ae986187 src/test/java/org/apache/aurora/scheduler/thrift/SchedulerThriftInterfaceTest.java 779dc302602ae8842084807ca868a491ea99b676 Diff: https://reviews.apache.org/r/51662/diff/ Testing --- Thanks, Karthik Anantha Padmanabhan
