Re: Review Request 51899: Ensure shell health checkers running for tasks running under an isolated fileystem are run within that filesystem.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51899/#review149099 --- Ship it! Ship It! src/main/python/apache/aurora/executor/common/health_checker.py (line 265) <https://reviews.apache.org/r/51899/#comment216613> nit on the name of `isolator`: `isolator` is already a well-defined concept within Mesos, and it seems to me that this is not related to that. Maybe consider naming this as `wrapped_fn`? - Zhitao Li On Sept. 15, 2016, 3:15 p.m., Joshua Cohen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51899/ > --- > > (Updated Sept. 15, 2016, 3:15 p.m.) > > > Review request for Aurora, Stephan Erb and Zhitao Li. > > > Repository: aurora > > > Description > --- > > Ensure shell health checkers running for tasks running under an isolated > fileystem are run within that filesystem. > > > Diffs > - > > src/main/python/apache/aurora/common/health_check/shell.py > 35750823553406a96282545066f1291c20347ffa > src/main/python/apache/aurora/executor/bin/thermos_executor_main.py > 5211f28e4e6c0efd29d7d79058128adb71ec7da8 > src/main/python/apache/aurora/executor/common/health_checker.py > 5fc845eceac6f0c048d7489fdc4c672b0c609ea0 > src/main/python/apache/thermos/common/BUILD > 879b812b6a262d6e13b64e662999dd436f039748 > src/main/python/apache/thermos/common/process_util.py PRE-CREATION > src/main/python/apache/thermos/core/process.py > 2134d4ff05861d4eaee9bc7ea4763e76ce63288c > src/test/python/apache/aurora/common/health_check/test_shell.py > 011464cbe1df00f2a56d4690176e7c2d0d3fd535 > src/test/python/apache/aurora/executor/common/test_health_checker.py > bb6ea69dd94298c5b8cf4d5f06d06eea7790d66e > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora > 290627f8bc38d31ae123cfd1cdd36e9291c2de18 > > Diff: https://reviews.apache.org/r/51899/diff/ > > > Testing > --- > > ./build-support/jenkins/build.sh > e2e tests > > > Thanks, > > Joshua Cohen > >
Re: Review Request 51746: Update e2e tests to verify running a docker image via the unified containerizer.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51746/#review148262 --- Ship it! Ship It! - Zhitao Li On Sept. 8, 2016, 8:56 p.m., Joshua Cohen wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51746/ > --- > > (Updated Sept. 8, 2016, 8:56 p.m.) > > > Review request for Aurora, Stephan Erb and Zhitao Li. > > > Repository: aurora > > > Description > --- > > Update e2e tests to verify running a docker image via the unified > containerizer. > > > Diffs > - > > examples/vagrant/mesos_config/etc_mesos-slave/docker_registry PRE-CREATION > examples/vagrant/mesos_config/etc_mesos-slave/image_providers > 7168c0ca4a81e0a1ce2320534015c6692b804b54 > examples/vagrant/mesos_config/etc_mesos-slave/isolation > c36223028513ace26040620b813022accaf0edf3 > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora > 042424d408a6b14be05c6e1c967215a53836d20e > src/test/sh/org/apache/aurora/e2e/http/http_example_bad_healthcheck.aurora > f80f2b1fe8accfa575b2c3e626feb8d926455f68 > src/test/sh/org/apache/aurora/e2e/http/http_example_updated.aurora > 4a95bc3a3d1ce50ea143266884efc4d0a7c6e767 > src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh > d81abe1f974bfb13b54a60b2018700aaf28a8ee6 > > Diff: https://reviews.apache.org/r/51746/diff/ > > > Testing > --- > > > Thanks, > > Joshua Cohen > >
Re: Review Request 51664: Document the Mesos containerizer
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51664/#review147908 --- docs/features/containers.md (lines 55 - 56) <https://reviews.apache.org/r/51664/#comment215206> Add a sentence to indicate that: `Otherwise, this user and its primary group has to exist in the image with matching uid/gip` - Zhitao Li On Sept. 6, 2016, 7:18 p.m., Stephan Erb wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51664/ > --- > > (Updated Sept. 6, 2016, 7:18 p.m.) > > > Review request for Aurora and Joshua Cohen. > > > Bugs: AURORA-1640 > https://issues.apache.org/jira/browse/AURORA-1640 > > > Repository: aurora > > > Description > --- > > Included changes: > > * enabled Docker support in our vagrant box > * consistent example jobs for both containerizers > * short enduser and operator documentation > * shuffled the reference documentation so that it is clear certain > limitations apply only to the Docker containerizer > > > Diffs > - > > docs/features/containers.md 6b9f717d672ef88be61341268f9c06923d217087 > docs/operations/configuration.md 85787b0815e9f0cb37c21efc04923b0e0bd10bf9 > docs/reference/configuration.md ff40262f1fb4eff780fbef17abcaecc457dc68d3 > examples/jobs/docker/hello_docker.aurora > d5611e6b78cb613c04e2fc3df54397ee4401a62e > examples/jobs/hello_docker_image.aurora PRE-CREATION > examples/vagrant/aurorabuild.sh 9daca067acc05c89ccc183e56bf30c787baf97dd > examples/vagrant/mesos_config/etc_mesos-slave/image_providers > 7168c0ca4a81e0a1ce2320534015c6692b804b54 > examples/vagrant/mesos_config/etc_mesos-slave/isolation > c36223028513ace26040620b813022accaf0edf3 > > Diff: https://reviews.apache.org/r/51664/diff/ > > > Testing > --- > > Rendered version available at > https://github.com/StephanErb/aurora/tree/containerizer-docs/docs > > > Thanks, > > Stephan Erb > >
Re: Review Request 51664: Document the Mesos containerizer
> On Sept. 6, 2016, 8 p.m., Zhitao Li wrote: > > docs/features/containers.md, lines 55-56 > > <https://reviews.apache.org/r/51664/diff/1/?file=1492106#file1492106line55> > > > > Add a sentence to indicate that: > > > > `Otherwise, this user and its primary group has to exist in the image > > with matching uid/gip` Sorry for typo, should be `uid/gid.` - Zhitao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51664/#review147908 --- On Sept. 6, 2016, 7:18 p.m., Stephan Erb wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51664/ > --- > > (Updated Sept. 6, 2016, 7:18 p.m.) > > > Review request for Aurora and Joshua Cohen. > > > Bugs: AURORA-1640 > https://issues.apache.org/jira/browse/AURORA-1640 > > > Repository: aurora > > > Description > --- > > Included changes: > > * enabled Docker support in our vagrant box > * consistent example jobs for both containerizers > * short enduser and operator documentation > * shuffled the reference documentation so that it is clear certain > limitations apply only to the Docker containerizer > > > Diffs > - > > docs/features/containers.md 6b9f717d672ef88be61341268f9c06923d217087 > docs/operations/configuration.md 85787b0815e9f0cb37c21efc04923b0e0bd10bf9 > docs/reference/configuration.md ff40262f1fb4eff780fbef17abcaecc457dc68d3 > examples/jobs/docker/hello_docker.aurora > d5611e6b78cb613c04e2fc3df54397ee4401a62e > examples/jobs/hello_docker_image.aurora PRE-CREATION > examples/vagrant/aurorabuild.sh 9daca067acc05c89ccc183e56bf30c787baf97dd > examples/vagrant/mesos_config/etc_mesos-slave/image_providers > 7168c0ca4a81e0a1ce2320534015c6692b804b54 > examples/vagrant/mesos_config/etc_mesos-slave/isolation > c36223028513ace26040620b813022accaf0edf3 > > Diff: https://reviews.apache.org/r/51664/diff/ > > > Testing > --- > > Rendered version available at > https://github.com/StephanErb/aurora/tree/containerizer-docs/docs > > > Thanks, > > Stephan Erb > >
Re: Review Request 51564: Allow E_NAME_IN_USE in useradd/groupadd.
> On Sept. 2, 2016, 6:35 p.m., John Sirois wrote: > > src/main/python/apache/aurora/executor/common/sandbox.py, line 251 > > <https://reviews.apache.org/r/51564/diff/6/?file=1490570#file1490570line251> > > > > You might extract this as a helper function > > (`_entity_exists(returncode: int): bool`) but definitely use a tuple > > instead of a list to represent the immutable set. Extracting just the set > > would do the trick, ~: > > ```python > > # Return codes from `useradd` and `groupadd` calls. > > _USER_OR_GROUP_EXISTS = ( > > 4, # uid/gid already exists. > > 9 # user/group name exists. > > ) > > ``` > > John Sirois wrote: > Zhitao - your call on this tweak. I'll patch this change in since Joshua > is away on travel and has handed over responsibility for this review to the 3 > of us who have shipited. Thanks for the review. For the sake of explicitness I'd like to keep them separate, and open the gate if we need to handle them separately. - Zhitao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51564/#review147681 --- On Sept. 2, 2016, 2:11 p.m., Zhitao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51564/ > --- > > (Updated Sept. 2, 2016, 2:11 p.m.) > > > Review request for Aurora, Joshua Cohen, John Sirois, and Zameer Manji. > > > Bugs: AURORA-1761 > https://issues.apache.org/jira/browse/AURORA-1761 > > > Repository: aurora > > > Description > --- > > Allow E_NAME_IN_USE in useradd/groupadd. > > > Diffs > - > > src/main/python/apache/aurora/executor/common/sandbox.py > a172691e164cf64792f65f049d698f9758336542 > src/test/python/apache/aurora/executor/common/test_sandbox.py > 57ab39e2444100c3a689bb0ff745c62f7bc2f1a6 > > Diff: https://reviews.apache.org/r/51564/diff/ > > > Testing > --- > > > Thanks, > > Zhitao Li > >
Re: Review Request 51564: Allow E_NAME_IN_USE in useradd/groupadd.
ight now is also possible if people use DockerContainerizer and specify a malicious `--volume` argument value. - Zhitao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51564/#review147497 --- On Aug. 31, 2016, 8:56 p.m., Zhitao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/51564/ > --- > > (Updated Aug. 31, 2016, 8:56 p.m.) > > > Review request for Aurora, Joshua Cohen, John Sirois, and Zameer Manji. > > > Bugs: AURORA-1761 > https://issues.apache.org/jira/browse/AURORA-1761 > > > Repository: aurora > > > Description > --- > > Allow E_NAME_IN_USE in useradd/groupadd. > > > Diffs > - > > src/main/python/apache/aurora/executor/common/sandbox.py > a172691e164cf64792f65f049d698f9758336542 > src/test/python/apache/aurora/executor/common/test_sandbox.py > 57ab39e2444100c3a689bb0ff745c62f7bc2f1a6 > > Diff: https://reviews.apache.org/r/51564/diff/ > > > Testing > --- > > > Thanks, > > Zhitao Li > >
Re: Review Request 51564: Allow E_NAME_IN_USE in useradd/groupadd.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51564/ --- (Updated Aug. 31, 2016, 8:56 p.m.) Review request for Aurora, Joshua Cohen, John Sirois, and Zameer Manji. Changes --- Comment line length fix. Bugs: AURORA-1761 https://issues.apache.org/jira/browse/AURORA-1761 Repository: aurora Description --- Allow E_NAME_IN_USE in useradd/groupadd. Diffs (updated) - src/main/python/apache/aurora/executor/common/sandbox.py a172691e164cf64792f65f049d698f9758336542 src/test/python/apache/aurora/executor/common/test_sandbox.py 57ab39e2444100c3a689bb0ff745c62f7bc2f1a6 Diff: https://reviews.apache.org/r/51564/diff/ Testing --- Thanks, Zhitao Li
Re: Review Request 51564: Allow E_NAME_IN_USE in useradd/groupadd.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/51564/ --- (Updated Aug. 31, 2016, 8:38 p.m.) Review request for Aurora, Joshua Cohen, John Sirois, and Zameer Manji. Changes --- Fix the test. Bugs: AURORA-1761 https://issues.apache.org/jira/browse/AURORA-1761 Repository: aurora Description --- Allow E_NAME_IN_USE in useradd/groupadd. Diffs (updated) - src/main/python/apache/aurora/executor/common/sandbox.py a172691e164cf64792f65f049d698f9758336542 src/test/python/apache/aurora/executor/common/test_sandbox.py 57ab39e2444100c3a689bb0ff745c62f7bc2f1a6 Diff: https://reviews.apache.org/r/51564/diff/ Testing --- Thanks, Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 5, 2016, 5:50 p.m., Maxim Khutornenko wrote:
> > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora, line 21
> > <https://reviews.apache.org/r/45177/diff/6/?file=1326033#file1326033line21>
> >
> > Curious, what do changes in this file actually test?
>
> Zhitao Li wrote:
> I used this to trigger the multiple ports case and test what SRV records
> Mesos DNS uses.
>
> I can revert change to this file if people don't like it, but I figured a
> slightly more complex example wouldn't really hurt.
>
> Maxim Khutornenko wrote:
> How did you test it? Any chance you could contribute an e2e test to back
> these changes up?
>
> Zhitao Li wrote:
> My testing was manual (see the json blobs in Testing Done of the review).
>
> I think it's actually possible to test this in e2e script:
> 1) enable this flag by default in vagrant;
> 2) curl the `/state` endpoint of master in the test_end_to_end.sh, parse
> the response through `jq` utility,
> 3) verify discovery part is not empty. (Including Mesos DNS would be way
> too heavy).
>
> The blocker is that `jq` is not installed by default in the vagrant box,
> so I can't really push such test safely.
>
> Maybe we can enable this by default in vagrant, and I can add the testing
> later after a virtualbox with jq is built?
>
> Maxim Khutornenko wrote:
> > The blocker is that jq is not installed by default in the vagrant box,
> so I can't really push such test safely.
>
> Can you pipe to `python -m json.tool` to get what you want instead? If
> that's not sufficient, I don't see much problem adding it to vbox image. See
> `build-support/packer/README.md` for instructions.
e2e test added. I'll send a oneline patch for installing jq, but I probably
need a committer who has persission to update the box to help for that.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review127147
---
On April 5, 2016, 11 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated April 5, 2016, 11 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Bugs: AURORA-1629
> https://issues.apache.org/jira/browse/AURORA-1629
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 2f935da01201cbef095e9d308630e187914f00ff
> docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
> docs/reference/scheduler-configuration.md
> e6c0bb60281d7c39c2aa235445ee4c1ef97464ba
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
> src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
> bb4fdec2fb2347635c6a7cc9e13133b4bf407771
> src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh
> 34717569365bbaad7117d7ec996e448462023fe8
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "location": "devcluster",
> "name": "http_example.test.vagrant",
> "ports": {
> "ports": [
> {
> "name": "tcp",
> "number": 31499
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 5, 2016, 5:50 p.m., Maxim Khutornenko wrote:
> > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora, line 21
> > <https://reviews.apache.org/r/45177/diff/6/?file=1326033#file1326033line21>
> >
> > Curious, what do changes in this file actually test?
>
> Zhitao Li wrote:
> I used this to trigger the multiple ports case and test what SRV records
> Mesos DNS uses.
>
> I can revert change to this file if people don't like it, but I figured a
> slightly more complex example wouldn't really hurt.
>
> Maxim Khutornenko wrote:
> How did you test it? Any chance you could contribute an e2e test to back
> these changes up?
My testing was manual (see the json blobs in Testing Done of the review).
I think it's actually possible to test this in e2e script:
1) enable this flag by default in vagrant;
2) curl the `/state` endpoint of master in the test_end_to_end.sh, parse the
response through `jq` utility,
3) verify discovery part is not empty. (Including Mesos DNS would be way too
heavy).
The blocker is that `jq` is not installed by default in the vagrant box, so I
can't really push such test safely.
Maybe we can enable this by default in vagrant, and I can add the testing later
after a virtualbox with jq is built?
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review127147
-------
On April 5, 2016, 6:15 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated April 5, 2016, 6:15 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Bugs: AURORA-1629
> https://issues.apache.org/jira/browse/AURORA-1629
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 2f935da01201cbef095e9d308630e187914f00ff
> docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
> docs/reference/scheduler-configuration.md
> e6c0bb60281d7c39c2aa235445ee4c1ef97464ba
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
> src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
> bb4fdec2fb2347635c6a7cc9e13133b4bf407771
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "location": "devcluster",
> "name": "http_example.test.vagrant",
> "ports": {
> "ports": [
> {
> "name": "tcp",
> "number": 31499,
> "protocol": "TCP"
> },
> {
> "name": "http",
> "number": 31529,
> "protocol": "TCP"
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
> "framework_id": "66d5ac1b-9243-4ef3-8016-40f19d079f5d-",
> "id": "vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31499-31499, 31529-31529]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated April 5, 2016, 6:15 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Changes
---
1. rebase;
2. make test more future proof by comparing full protobuf object instead;
3. document and typo fixes.
Bugs: AURORA-1629
https://issues.apache.org/jira/browse/AURORA-1629
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 2f935da01201cbef095e9d308630e187914f00ff
docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
docs/reference/scheduler-configuration.md
e6c0bb60281d7c39c2aa235445ee4c1ef97464ba
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
bb4fdec2fb2347635c6a7cc9e13133b4bf407771
Diff: https://reviews.apache.org/r/45177/diff/
Testing
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"location": "devcluster",
"name": "http_example.test.vagrant",
"ports": {
"ports": [
{
"name": "tcp",
"number": 31499,
"protocol": "TCP"
},
{
"name": "http",
"number": 31529,
"protocol": "TCP"
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"framework_id": "66d5ac1b-9243-4ef3-8016-40f19d079f5d-",
"id": "vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31499-31499, 31529-31529]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 5, 2016, 5:50 p.m., Maxim Khutornenko wrote:
> > src/test/sh/org/apache/aurora/e2e/http/http_example.aurora, line 21
> > <https://reviews.apache.org/r/45177/diff/6/?file=1326033#file1326033line21>
> >
> > Curious, what do changes in this file actually test?
I used this to trigger the multiple ports case and test what SRV records Mesos
DNS uses.
I can revert change to this file if people don't like it, but I figured a
slightly more complex example wouldn't really hurt.
> On April 5, 2016, 5:50 p.m., Maxim Khutornenko wrote:
> > docs/features/service-discovery.md, line 31
> > <https://reviews.apache.org/r/45177/diff/6/?file=1326025#file1326025line31>
> >
> > Where is `twitterscheduler` part coming from in this example?
This is the lower case of
`CommandLineDriverSettingsModule.TWITTER_FRAMEWORK_NAME`. There is a TODO to
allow this being reconfigured in Aurora, and I also filed an issue to Mesos DNS.
I'll add some explanation.
> On April 5, 2016, 5:50 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line
> > 252
> > <https://reviews.apache.org/r/45177/diff/6/?file=1326030#file1326030line252>
> >
> > This should be moved above the previous line it belongs to.
fixing
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review127147
---
On April 5, 2016, 4:20 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated April 5, 2016, 4:20 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Bugs: AURORA-1629
> https://issues.apache.org/jira/browse/AURORA-1629
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 1df0345fac6ec6da92be15eb246c6957dea066bf
> docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
> docs/reference/scheduler-configuration.md
> e6c0bb60281d7c39c2aa235445ee4c1ef97464ba
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
> src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
> bb4fdec2fb2347635c6a7cc9e13133b4bf407771
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "location": "devcluster",
> "name": "http_example.test.vagrant",
> "ports": {
> "ports": [
> {
> "name": "tcp",
> "number": 31499,
> "protocol": "TCP"
> },
> {
> "name": "http",
> "number": 31529,
> "protocol": "TCP"
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
> "framework_id": "66d5ac1b-9243-4ef3-8016-40f19d079f5d-",
> "id": "vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31499-31499, 31529-31529]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated April 5, 2016, 4:20 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Changes
---
rebase
Bugs: AURORA-1629
https://issues.apache.org/jira/browse/AURORA-1629
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 1df0345fac6ec6da92be15eb246c6957dea066bf
docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
docs/reference/scheduler-configuration.md
e6c0bb60281d7c39c2aa235445ee4c1ef97464ba
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
bb4fdec2fb2347635c6a7cc9e13133b4bf407771
Diff: https://reviews.apache.org/r/45177/diff/
Testing
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"location": "devcluster",
"name": "http_example.test.vagrant",
"ports": {
"ports": [
{
"name": "tcp",
"number": 31499,
"protocol": "TCP"
},
{
"name": "http",
"number": 31529,
"protocol": "TCP"
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"framework_id": "66d5ac1b-9243-4ef3-8016-40f19d079f5d-",
"id": "vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31499-31499, 31529-31529]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated April 5, 2016, 4:02 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Changes
---
1. Change http_example to multiple ports;
2. Add documentation for how multiple ports work for Mesos DNS, and other
limitation;
3. New flag doc in scheduler-configurtion.md.
Bugs: AURORA-1629
https://issues.apache.org/jira/browse/AURORA-1629
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
docs/reference/scheduler-configuration.md
9945531aeb53db7387e3b030521be94a66a0c9f0
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
src/test/sh/org/apache/aurora/e2e/http/http_example.aurora
bb4fdec2fb2347635c6a7cc9e13133b4bf407771
Diff: https://reviews.apache.org/r/45177/diff/
Testing (updated)
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"location": "devcluster",
"name": "http_example.test.vagrant",
"ports": {
"ports": [
{
"name": "tcp",
"number": 31499,
"protocol": "TCP"
},
{
"name": "http",
"number": 31529,
"protocol": "TCP"
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"framework_id": "66d5ac1b-9243-4ef3-8016-40f19d079f5d-",
"id": "vagrant-test-http_example-1-e56d2834-3df3-4d6e-9399-4f5e80668703",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31499-31499, 31529-31529]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 5, 2016, 1:39 p.m., Stephan Erb wrote:
> > docs/features/service-discovery.md, line 31
> > <https://reviews.apache.org/r/45177/diff/4/?file=1324845#file1324845line31>
> >
> > How does Mesos DNS find out about our primary port? The code below
> > reads like we just populate all ports.
Performed more testing: in fact Mesos DNS merges all (ip, port) tuples into
this one record.
It has another (pretty undocumented) SRV record of form
`_{port-name}.{name}.{framework}.{domain}`, which should be used for multiple
ports case instead.
I'm updating document to reflect this.
> On April 5, 2016, 1:39 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java,
> > line 106
> > <https://reviews.apache.org/r/45177/diff/4/?file=1324847#file1324847line106>
> >
> > Please add this flag to the `scheduler-configuration.md` documentation
> > (we should really automate that one day...)
Will do.
> On April 5, 2016, 1:39 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line
> > 253
> > <https://reviews.apache.org/r/45177/diff/4/?file=1324849#file1324849line253>
> >
> > This has the disadvantage that it will ignore the portmap of the
> > announcer. The `mname` feature has the same problem. The portmap resolve
> > code does currently only live in python.
> > https://github.com/apache/aurora/blob/b24619b28c4dbb35188871bacd0091a9e01218e3/src/main/python/apache/aurora/config/port_resolver.py#L22
> >
> >
> > I don't think the limitation is very severe right now (given that this
> > is still a beta feature).
> > However you should probably be aware of that. Maybe you could mention
> > it in the documentation.
Will mention this in document.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review126960
---
On April 5, 2016, 1:50 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated April 5, 2016, 1:50 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Bugs: AURORA-1629
> https://issues.apache.org/jira/browse/AURORA-1629
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
> docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "name": "vagrant.test.http_example.1",
> "ports": {
> "ports": [
> {
> "name": "http",
> "number": 31648
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
> "id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31648-31648]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated April 5, 2016, 3:50 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Bugs: AURORA-1629
https://issues.apache.org/jira/browse/AURORA-1629
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs
-
RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
Diff: https://reviews.apache.org/r/45177/diff/
Testing
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"name": "vagrant.test.http_example.1",
"ports": {
"ports": [
{
"name": "http",
"number": 31648
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
"id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31648-31648]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 1, 2016, 12:05 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line
> > 242
> > <https://reviews.apache.org/r/45177/diff/3/?file=1321463#file1321463line242>
> >
> > Should we consider using the inverse notation here? This would be
> > `...` instead of
> > `...`? @benley made a simmilar
> > comment on the mailing list.
> >
> > What do you think?
>
> Zhitao Li wrote:
> I'll give this a try and test again, but I suspect current implementation
> of Mesos DNS simply swallows dot and truncates the string with its own
> standard.
>
> `` probably should be left out by default since client wants
> load balance among instances instead of one instance per record?
Sorry I was wrong in previous comment, inverse DNS record actualy looks pretty
nice (twitterscheduler is still there, but I don't think it's a big problem to
myself, especially given last `mesos` domain is customizable in Mesos DNS).
Updated code and doc.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review126574
---
On April 4, 2016, 7:44 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated April 4, 2016, 7:44 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
> docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "name": "vagrant.test.http_example.1",
> "ports": {
> "ports": [
> {
> "name": "http",
> "number": 31648
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
> "id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31648-31648]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated April 4, 2016, 7:44 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Changes
---
Inverse DNS name, and doc update as requested.
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
docs/features/service-discovery.md 858ca2a137db9c8fbce20e9b2b924447fe7a41ae
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
Diff: https://reviews.apache.org/r/45177/diff/
Testing
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"name": "vagrant.test.http_example.1",
"ports": {
"ports": [
{
"name": "http",
"number": 31648
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
"id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31648-31648]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On April 1, 2016, 12:05 p.m., Stephan Erb wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line
> > 242
> > <https://reviews.apache.org/r/45177/diff/3/?file=1321463#file1321463line242>
> >
> > Should we consider using the inverse notation here? This would be
> > `...` instead of
> > `...`? @benley made a simmilar
> > comment on the mailing list.
> >
> > What do you think?
I'll give this a try and test again, but I suspect current implementation of
Mesos DNS simply swallows dot and truncates the string with its own standard.
`` probably should be left out by default since client wants load
balance among instances instead of one instance per record?
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review126574
-----------
On March 31, 2016, 9:48 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated March 31, 2016, 9:48 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "name": "vagrant.test.http_example.1",
> "ports": {
> "ports": [
> {
> "name": "http",
> "number": 31648
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
> "id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31648-31648]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
> On March 31, 2016, 8:25 p.m., Joshua Cohen wrote:
> > RELEASE-NOTES.md, line 14
> > <https://reviews.apache.org/r/45177/diff/2/?file=1321194#file1321194line14>
> >
> > Is there any reason we need a command line arg to control this? Is
> > there any detriment to just always populating `DiscoveryInfo`?
>
> Zhitao Li wrote:
> The only possible concern I have for now is that this would 1) generate
> more data to Mesos and master/slave might have higher memory pressure, and 2)
> Mesos endpoints like /state.json might be slower.
>
> Giving operator a flag to control this would allow people to test it out.
>
> Unfortunately, I don't have a large Aurora cluster running so I can't
> really quantifiy it's impact right now.
>
> Joshua Cohen wrote:
> In my gut I feel like it shouldn't be a problem, but out of an abundance
> of caution, let's go ahead with the flag. We can remove it in the future if
> the functionality is deemed harmless to large clusters.
Agreed. Removing it in the future is exactly what I planned.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review126428
-------
On March 31, 2016, 9:48 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated March 31, 2016, 9:48 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
> examples/vagrant/upstart/aurora-scheduler.conf
> d61801c0b7c9683434a548a26bed3c49bbb75927
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> add1270409b7e82578267a2c10cf2afe6e93e3de
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 85c550bb5754154de5f639df4a0992d10cb70993
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 3a6048633efceec68b15d3c511237c0f1871a3fc
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "name": "vagrant.test.http_example.1",
> "ports": {
> "ports": [
> {
> "name": "http",
> "number": 31648
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
> "id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31648-31648]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated March 31, 2016, 9:48 p.m.)
Review request for Aurora, Kunal Thakar and Stephan Erb.
Changes
---
Rebase and add environment and location to DiscoveryInfo as requested by Joshua.
Repository: aurora
Description
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 6fc3afeb5a9e2f2c2ba944fbc6d611d3494eb779
examples/vagrant/upstart/aurora-scheduler.conf
d61801c0b7c9683434a548a26bed3c49bbb75927
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
add1270409b7e82578267a2c10cf2afe6e93e3de
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
85c550bb5754154de5f639df4a0992d10cb70993
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
7110fbd78b1b4356acd40e5ab9c3379e5ac19df7
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
3a6048633efceec68b15d3c511237c0f1871a3fc
Diff: https://reviews.apache.org/r/45177/diff/
Testing
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"name": "vagrant.test.http_example.1",
"ports": {
"ports": [
{
"name": "http",
"number": 31648
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
"id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31648-31648]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 45177: Setting DiscoveryInfo.
> On March 31, 2016, 8:25 p.m., Joshua Cohen wrote:
> > RELEASE-NOTES.md, line 14
> > <https://reviews.apache.org/r/45177/diff/2/?file=1321194#file1321194line14>
> >
> > Is there any reason we need a command line arg to control this? Is
> > there any detriment to just always populating `DiscoveryInfo`?
The only possible concern I have for now is that this would 1) generate more
data to Mesos and master/slave might have higher memory pressure, and 2) Mesos
endpoints like /state.json might be slower.
Giving operator a flag to control this would allow people to test it out.
Unfortunately, I don't have a large Aurora cluster running so I can't really
quantifiy it's impact right now.
> On March 31, 2016, 8:25 p.m., Joshua Cohen wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java, line
> > 242
> > <https://reviews.apache.org/r/45177/diff/2/?file=1321198#file1321198line242>
> >
> > Can you also set DiscoveryInfo.location to the cluster name? From the
> > docs on DiscoveryInfo in
> > [mesos.proto](https://github.com/apache/mesos/blob/master/include/mesos/mesos.proto#L1831-L1836):
> >
> > the location field may receive values like
> > EAST-US/WEST-US/EUROPE/AMEA
> >
> > This seems well aligned with our cluster?
Yeah I wanted to do this. Do you know how to get that information inside Aurora
scheduler process?
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/#review126428
---
On March 31, 2016, 7:52 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/45177/
> ---
>
> (Updated March 31, 2016, 7:52 p.m.)
>
>
> Review request for Aurora, Kunal Thakar and Stephan Erb.
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This allows alternative service discovery methodologies to find tasks from
> Aurora (e.g. mesos-dns), especially the dynamic port mapping.
>
>
> Diffs
> -
>
> RELEASE-NOTES.md 34f28a165aae4ae24fa95ef19b4972e088fd63a0
> examples/vagrant/upstart/aurora-scheduler.conf
> 120b89a1dc10a259940cb9527eb2517f19d04471
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
> 949c299bdbc54f976db994266fb97f3099256f13
>
> src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
> 7beea81a1e97f3f2fce215ff3e364919c3aebf6d
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 20cbd419dcb988f2c7ba72c8acbe6fee90d02248
> src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
> 16fa2d35cdf7ecdf82dd1ec7739e685ec1ff1aa2
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> 5103bd0f43d53079976b0f1596e299f2d91aa860
>
> Diff: https://reviews.apache.org/r/45177/diff/
>
>
> Testing
> ---
>
> 1. Added unit test;
> 2. Use the vagrant environment to start the example in http_example.aurora,
> and observe the following in master's state.json:
>
> ```
> curl 192.168.33.7:5050/state | jq . | less
>
> "tasks": [
> {
> "discovery": {
> "environment": "test",
> "name": "vagrant.test.http_example.1",
> "ports": {
> "ports": [
> {
> "name": "http",
> "number": 31648
> }
> ]
> },
> "visibility": "CLUSTER"
> },
> "executor_id":
> "thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
> "id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
> "name": "vagrant/test/http_example",
> "resources": {
> "cpus": 0.4,
> "disk": 64,
> "mem": 32,
> "ports": "[31648-31648]"
> },
> ...
>
> ```
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 45177: Setting DiscoveryInfo.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/45177/
---
(Updated March 31, 2016, 7:51 p.m.)
Review request for Aurora.
Changes
---
Unit test and command line flag switch.
Summary (updated)
-
Setting DiscoveryInfo.
Repository: aurora
Description (updated)
---
This allows alternative service discovery methodologies to find tasks from
Aurora (e.g. mesos-dns), especially the dynamic port mapping.
Diffs (updated)
-
RELEASE-NOTES.md 34f28a165aae4ae24fa95ef19b4972e088fd63a0
examples/vagrant/upstart/aurora-scheduler.conf
120b89a1dc10a259940cb9527eb2517f19d04471
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java
949c299bdbc54f976db994266fb97f3099256f13
src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java
7beea81a1e97f3f2fce215ff3e364919c3aebf6d
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
20cbd419dcb988f2c7ba72c8acbe6fee90d02248
src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java
16fa2d35cdf7ecdf82dd1ec7739e685ec1ff1aa2
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
5103bd0f43d53079976b0f1596e299f2d91aa860
Diff: https://reviews.apache.org/r/45177/diff/
Testing (updated)
---
1. Added unit test;
2. Use the vagrant environment to start the example in http_example.aurora, and
observe the following in master's state.json:
```
curl 192.168.33.7:5050/state | jq . | less
"tasks": [
{
"discovery": {
"environment": "test",
"name": "vagrant.test.http_example.1",
"ports": {
"ports": [
{
"name": "http",
"number": 31648
}
]
},
"visibility": "CLUSTER"
},
"executor_id":
"thermos-vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"framework_id": "770e194c-6366-41f2-8a2d-e2c616aa9490-",
"id": "vagrant-test-http_example-1-69c3908a-e8cd-44a7-a824-dd87ef5920f0",
"name": "vagrant/test/http_example",
"resources": {
"cpus": 0.4,
"disk": 64,
"mem": 32,
"ports": "[31648-31648]"
},
...
```
Thanks,
Zhitao Li
Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/44685/#review123052 --- nit: I feel like this diff may be better described as "Add support for running docker tasks without (thermos) executor", unless there is additional work planned in same diff. - Zhitao Li On March 11, 2016, 12:44 a.m., Bill Farner wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/44685/ > --- > > (Updated March 11, 2016, 12:44 a.m.) > > > Review request for Aurora, Joshua Cohen and John Sirois. > > > Repository: aurora > > > Description > --- > > This is currently labeled as experimental. > > Only the most basic wiring is added here, and assumes that the provided image > includes an ENTRYPOINT. Unlike Docker support via the thermos executor, this > approach allows containers with an entrypoint, and does not impose environment > requirements on the image (e.g. python interpreter, libmesos dependencies). > > Note that when using this, other familiar Aurora facilities that relate to the > thermos executor will not work. For example, browsing task logs is not > supported. > > Support for exercising this from the client will come shortly. > > > Diffs > - > > NEWS 0aa7f5e192ef17b95471e34a5408491999bcdeeb > src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java > 60746383fccb107ca27925a91aa1803e2cf0fd85 > src/main/java/org/apache/aurora/scheduler/app/AppModule.java > a0d2a717534bbb2e85a556721cc53c1e4b743461 > src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java > 1de6966565d2fbd9abd220ad8162b624b109959a > > src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java > 6300e5f83b039a9798e7093f6b46c84566e507e0 > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java > a34af4d2fb3863ab8197bcdce942c513d629621b > > src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java > d2789d0eaaeba99fcff3412f1abdd29a09d6514d > > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java > 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 > > Diff: https://reviews.apache.org/r/44685/diff/ > > > Testing > --- > > Via additional hacking, i successfully ran the stock [hello > world](https://hub.docker.com/_/hello-world/) image. Within the sandbox, i > observed the expected output in the `stdout` file. Status updates for the > task exiting worked as expected. > > > Thanks, > > Bill Farner > >
Re: Review Request 43112: Make --announcer-enable optional no-op instead of removing it completely.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43112/ --- (Updated Feb. 2, 2016, 10:13 p.m.) Review request for Aurora, Kunal Thakar, Bill Farner, and Zameer Manji. Changes --- Rebase to pick up NEWS change. Repository: aurora Description --- Make --announcer-enable optional no-op instead of removing it completely. For easy upgrade purpose. Diffs (updated) - NEWS 318979ed5e408e42df0c5a77c35a653f14bc7233 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f4f5cd77b6444c225ec960c7e2cf5349a80bd344 Diff: https://reviews.apache.org/r/43112/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor:executor Thanks, Zhitao Li
Re: Review Request 43112: Make --announcer-enable optional no-op instead of removing it completely.
> On Feb. 2, 2016, 10:08 p.m., Bill Farner wrote: > > Ship It! > > Bill Farner wrote: > Please also file a ticket to do the actual removal, make it a blocker for > https://issues.apache.org/jira/browse/AURORA-1586 https://issues.apache.org/jira/browse/AURORA-1606 - Zhitao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43112/#review117494 --- On Feb. 2, 2016, 10:13 p.m., Zhitao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/43112/ > --- > > (Updated Feb. 2, 2016, 10:13 p.m.) > > > Review request for Aurora, Kunal Thakar, Bill Farner, and Zameer Manji. > > > Repository: aurora > > > Description > --- > > Make --announcer-enable optional no-op instead of removing it completely. > > For easy upgrade purpose. > > > Diffs > - > > NEWS 318979ed5e408e42df0c5a77c35a653f14bc7233 > src/main/python/apache/aurora/executor/bin/thermos_executor_main.py > f4f5cd77b6444c225ec960c7e2cf5349a80bd344 > > Diff: https://reviews.apache.org/r/43112/diff/ > > > Testing > --- > > ./pants test src/test/python/apache/aurora/executor:executor > > > Thanks, > > Zhitao Li > >
Re: Review Request 42727: Remove the --announcer-enable executor flag.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42727/#review117482 --- Sorry for coming to this only after it's committed, but I think this might create trouble for operators during upgrade. Because the new executor will not understand the `--announce-enable` flag and crashes, the operator has to coordinate disabling the flag from scheduler side first, and push executor. Also, during this window between scheduler upgrade and cluster-wise executor upgrade, announce might be broken. Can I suggest to make this flag optional and no-op for the deprecation cycle (max of one version or 90 days according to recent email conversations), then remove this afterwards? - Zhitao Li On Jan. 26, 2016, 6:40 p.m., Bill Farner wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42727/ > --- > > (Updated Jan. 26, 2016, 6:40 p.m.) > > > Review request for Aurora, Maxim Khutornenko and Zameer Manji. > > > Repository: aurora > > > Description > --- > > Remove the --announcer-enable executor flag. > > > Diffs > - > > NEWS 3f40aba46dc72b50607c82a2cc89040b1d10048a > examples/vagrant/upstart/aurora-scheduler-kerberos.conf > 57fa312ca6c007731fd88ace1bea507320d14700 > examples/vagrant/upstart/aurora-scheduler.conf > 69025dd85011751b1036615af9944a4f7693bb15 > src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java > 6214882cd60211b18fa4ce6f43442184b2dccac8 > src/main/python/apache/aurora/executor/bin/thermos_executor_main.py > 12726936ee85d2ee3dba19eb497873d7422886e6 > > src/test/resources/org/apache/aurora/scheduler/configuration/executor/test-thermos-executor.json > 8d4d90b18451d8a2cc7cbe2d25f64942d0045491 > > Diff: https://reviews.apache.org/r/42727/diff/ > > > Testing > --- > > Running tests now. > > > Thanks, > > Bill Farner > >
Review Request 43112: Make --announcer-enable optional no-op instead of removing it completely.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43112/ --- Review request for Aurora, Kunal Thakar, Bill Farner, and Zameer Manji. Repository: aurora Description --- Make --announcer-enable optional no-op instead of removing it completely. For easy upgrade purpose. Diffs - NEWS b1713602c2531ac39fc97c2044e024c21808ad63 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f4f5cd77b6444c225ec960c7e2cf5349a80bd344 Diff: https://reviews.apache.org/r/43112/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor:executor Thanks, Zhitao Li
Re: Review Request 42727: Remove the --announcer-enable executor flag.
> On Feb. 2, 2016, 9:31 p.m., Zhitao Li wrote: > > Sorry for coming to this only after it's committed, but I think this might > > create trouble for operators during upgrade. > > > > Because the new executor will not understand the `--announce-enable` flag > > and crashes, the operator has to coordinate disabling the flag from > > scheduler side first, and push executor. Also, during this window between > > scheduler upgrade and cluster-wise executor upgrade, announce might be > > broken. > > > > Can I suggest to make this flag optional and no-op for the deprecation > > cycle (max of one version or 90 days according to recent email > > conversations), then remove this afterwards? > > Bill Farner wrote: > You could work around this with a script that wraps the executor, but > that would be overly-complicated. Your suggestion sounds spot on. I'm not > available at the moment to put together the patch, but i can review/ship! https://reviews.apache.org/r/43112/ includes a quick patch. - Zhitao --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42727/#review117482 --- On Jan. 26, 2016, 6:40 p.m., Bill Farner wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42727/ > --- > > (Updated Jan. 26, 2016, 6:40 p.m.) > > > Review request for Aurora, Maxim Khutornenko and Zameer Manji. > > > Repository: aurora > > > Description > --- > > Remove the --announcer-enable executor flag. > > > Diffs > - > > NEWS 3f40aba46dc72b50607c82a2cc89040b1d10048a > examples/vagrant/upstart/aurora-scheduler-kerberos.conf > 57fa312ca6c007731fd88ace1bea507320d14700 > examples/vagrant/upstart/aurora-scheduler.conf > 69025dd85011751b1036615af9944a4f7693bb15 > src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java > 6214882cd60211b18fa4ce6f43442184b2dccac8 > src/main/python/apache/aurora/executor/bin/thermos_executor_main.py > 12726936ee85d2ee3dba19eb497873d7422886e6 > > src/test/resources/org/apache/aurora/scheduler/configuration/executor/test-thermos-executor.json > 8d4d90b18451d8a2cc7cbe2d25f64942d0045491 > > Diff: https://reviews.apache.org/r/42727/diff/ > > > Testing > --- > > Running tests now. > > > Thanks, > > Bill Farner > >
Re: Review Request 43112: Make --announcer-enable optional no-op instead of removing it completely.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/43112/ --- (Updated Feb. 2, 2016, 11:24 p.m.) Review request for Aurora, Kunal Thakar, Bill Farner, and Zameer Manji. Changes --- Put issue number in TODO. Repository: aurora Description --- Make --announcer-enable optional no-op instead of removing it completely. For easy upgrade purpose. Diffs (updated) - NEWS 318979ed5e408e42df0c5a77c35a653f14bc7233 src/main/python/apache/aurora/executor/bin/thermos_executor_main.py f4f5cd77b6444c225ec960c7e2cf5349a80bd344 Diff: https://reviews.apache.org/r/43112/diff/ Testing --- ./pants test src/test/python/apache/aurora/executor:executor Thanks, Zhitao Li
Re: Review Request 35990: Map Aurora task metadata to Mesos task labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35990/#review116637 --- Ship it! NEWS (line 45) <https://reviews.apache.org/r/35990/#comment177706> Maybe mention the constant label prefix? - Zhitao Li On Jan. 27, 2016, 10:17 p.m., Stephan Erb wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35990/ > --- > > (Updated Jan. 27, 2016, 10:17 p.m.) > > > Review request for Aurora, Bill Farner, Zhitao Li, and Zameer Manji. > > > Bugs: AURORA-1052 > https://issues.apache.org/jira/browse/AURORA-1052 > > > Repository: aurora > > > Description > --- > > Map Aurora task metadata to Mesos task labels > > > Diffs > - > > NEWS 29702d581bced316b313c5caa350de975a8bf428 > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java > fcad0e735b676e8da9b2b1d4d0d5e734a717b2d8 > > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java > cc2a4155008672056410086b05b6cc875485f669 > > Diff: https://reviews.apache.org/r/35990/diff/ > > > Testing > --- > > ./gradlew -Pq build > > > Thanks, > > Stephan Erb > >
Re: Review Request 35990: Map Aurora task metadata to Mesos task labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35990/#review116647 --- Ship it! Ship It! - Zhitao Li On Jan. 27, 2016, 10:52 p.m., Stephan Erb wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35990/ > --- > > (Updated Jan. 27, 2016, 10:52 p.m.) > > > Review request for Aurora, Bill Farner, Zhitao Li, and Zameer Manji. > > > Bugs: AURORA-1052 > https://issues.apache.org/jira/browse/AURORA-1052 > > > Repository: aurora > > > Description > --- > > Map Aurora task metadata to Mesos task labels > > > Diffs > - > > NEWS 29702d581bced316b313c5caa350de975a8bf428 > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java > fcad0e735b676e8da9b2b1d4d0d5e734a717b2d8 > > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java > cc2a4155008672056410086b05b6cc875485f669 > > Diff: https://reviews.apache.org/r/35990/diff/ > > > Testing > --- > > ./gradlew -Pq build > > > Thanks, > > Stephan Erb > >
Re: Review Request 35990: Map Aurora task metadata to Mesos task labels
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/35990/#review116639 --- src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java (line 77) <https://reviews.apache.org/r/35990/#comment177708> Hmm, I think this is missing a trailing dot. - Zhitao Li On Jan. 27, 2016, 10:17 p.m., Stephan Erb wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/35990/ > --- > > (Updated Jan. 27, 2016, 10:17 p.m.) > > > Review request for Aurora, Bill Farner, Zhitao Li, and Zameer Manji. > > > Bugs: AURORA-1052 > https://issues.apache.org/jira/browse/AURORA-1052 > > > Repository: aurora > > > Description > --- > > Map Aurora task metadata to Mesos task labels > > > Diffs > - > > NEWS 29702d581bced316b313c5caa350de975a8bf428 > src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java > fcad0e735b676e8da9b2b1d4d0d5e734a717b2d8 > > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java > cc2a4155008672056410086b05b6cc875485f669 > > Diff: https://reviews.apache.org/r/35990/diff/ > > > Testing > --- > > ./gradlew -Pq build > > > Thanks, > > Stephan Erb > >
Re: Review Request 42177: Vagrant change to reserve part of the dev cluster's resources to 'aurora-role'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42177/ --- (Updated Jan. 19, 2016, 6:46 p.m.) Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill Farner. Changes --- Revert change to log location and rebase with master. Bugs: AURORA-1109 https://issues.apache.org/jira/browse/AURORA-1109 Repository: aurora Description --- This is how I tested changes in https://reviews.apache.org/r/42176/. Diffs (updated) - examples/vagrant/upstart/aurora-scheduler.conf 8120ff7706f0ad0810ec9018c55d0d065066cbc5 examples/vagrant/upstart/mesos-master.conf 9d7491c08e14e3951b2fd2f74291a2009883c379 examples/vagrant/upstart/mesos-slave.conf 1ef059b17d16d4f1594a19ff6422ea653a413359 Diff: https://reviews.apache.org/r/42177/diff/ Testing --- Integration test script. Thanks, Zhitao Li
Re: Review Request 42126: Accept resource offers from multiple framework roles.
> On Jan. 14, 2016, 2:45 p.m., Bill Farner wrote:
> > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java,
> > line 216
> > <https://reviews.apache.org/r/42126/diff/9/?file=1196221#file1196221line216>
> >
> > I assume this line will be restored once `AcceptedOffer` accommodates
> > for floating point errors?
Yes, with a slightly different `Offer` object which uses `ResourceSlot` from
`SOME_OVERHEAD_EXECUTOR` instead of `THERMOS_EXECUTOR`.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114478
-------
On Jan. 14, 2016, 2:46 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 14, 2016, 2:46 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: Accept resource offers from multiple framework roles.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 14, 2016, 6:23 p.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Clearer helper message as requested by Maxim
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: Accept resource offers from multiple framework roles.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 14, 2016, 6:46 a.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Tweaked review summary in preparation for commit.
-wfarner
Summary (updated)
-
Accept resource offers from multiple framework roles.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs
-
NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: Accept resource offers from multiple framework roles.
> On Jan. 14, 2016, 1:06 a.m., Dmitriy Shirchenko wrote: > > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java, > > lines 216-217 > > <https://reviews.apache.org/r/42126/diff/8/?file=1196146#file1196146line216> > > > > Is this intentionally commented out? > > Zhitao Li wrote: > Yes this is intentially removed. Old behavior of changing this `config` > here resulted in `resources` in static final object `OFFER` not sufficient to > be allocated, and I don't see any point to use a different executor for this > test case, so I've deleted this code. > > If any reviewer thinks there is a reason to test this with a different > executor, I can reconstruct an `Offer` object with correct resource and add > this back. > > Bill Farner wrote: > This is slightly suspicious...are you sure resource accounting wasn't > impacted in some way? I would expect old test cases to logically work with > this feature. > > Zhitao Li wrote: > I recalled why I commented this out in the first pass now (this was a bit > tricky and I mistakenly thought it's logical bug while it's due to floating > point precision). > > In TaskExecutors.java, `SOME_OVERHEAD_EXECUTOR` uses `0.01` cpus: > ``` > public static final ExecutorSettings SOME_OVERHEAD_EXECUTOR = > TestExecutorSettings.thermosOnlyWithOverhead( > new ResourceSlot(0.01, Amount.of(256L, Data.MB), Amount.of(0L, > Data.MB), 0)); > ``` > Unfortunately, `5.01 - 5 - 0.01` **is not zero** in floating point math > of Java, so the test keeps throwing `InsufficientResourcesException` even if > I constructed an `Offer` whose resource is correctly equivalent to > `ResourceSlot.from(TASK_CONFIG).add(TaskExecutors.SOME_OVERHEAD_EXECUTOR.getExecutorOverhead())`. > > **My proposal is to change all comparison with zero in `AcceptedOffer` > class (3 places in current patch) to compare with an `EPSILON` whose value is > `1e-6` or something even smaller.** I don't think there is any reasonable > usage of resource unit smaller than this granularity, and the floating point > math in the class would be more robust. We can document this behavior > somewhere if it sounds interesting. > > The default resource usage for `THERMOS_EXECUTOR` is 0.25 so I don't > encounter this elsewhere. > > Some alternatives which I liked less: > 1. Change `TaskExecutors.SOME_OVERHEAD_EXECUTOR`'s cpus to a value which > could be precisely represented by floating point (0.125 or something similar); > 2. Manually add a bit more cpu resource to the `Offer` object to trick > the test to pass. > > If the above proposed solution sounds fine, I can send an update to the > patch. > > Bill Farner wrote: > I'm in favor of using an EPSILON value. Please be sure to extract a > function to handle the equivalence check. > > It's too bad mesos doesn't use fixed point for scalar resources to avoid > these issues. Looks like there's been recent momentum to make that change: > https://issues.apache.org/jira/browse/MESOS-3997 > > They recently addressed an instance of this issue with this change: > > https://github.com/apache/mesos/commit/5b5dd566aea71f654c51b2706a958ca1b5cb07a3 > > ``` > CHECK_NEAR(result.cpus().get(), cpus().get(), MIN_CPUS); > ``` > > For reference, `MIN_CPUS` is 0.01. Will do with a `nearZero()` helper function. For the value of `ESPILON`, I took a quick look at related Mesos code, and the minimum gradunalar which could make a difference is `MIN_CPUS` / `CPU_SHARES_PER_CPU` (1024) in cgroup based isolation, which is still larger than `1e-6`, so using the latter value should ensure Aurora user won't lose precision even at extreme scheduling condition. - Zhitao ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42126/#review114353 --- On Jan. 14, 2016, 2:46 p.m., Zhitao Li wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/42126/ > --- > > (Updated Jan. 14, 2016, 2:46 p.m.) > > > Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill > Farner. > > > Bugs: AURORA-1109 > https://issues.apache.org/jira/browse/AURORA-1109 > > > Repository: aurora > > > Description > --- > > This review is a prototype for introducing mu
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 13, 2016, 11:01 p.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java, line 79
> > <https://reviews.apache.org/r/42126/diff/5/?file=1195816#file1195816line79>
> >
> > I'd expect this value populated from the command line arg supplied to
> > driver setting module. Mind explaining the logic here? Why do we default to
> > "*"?
>
> Bill Farner wrote:
> Is this not necessary to align with the special handling of `*` on the
> mesos side?
>
> Maxim Khutornenko wrote:
> I guess my question was about what this default even means. I'd expect
> this feature to be OFF by default by not registering with any role at all. It
> should be up to cluster operator to decide what role to register with (if at
> all).
>
> Bill Farner wrote:
> In this context, i believe the logic is appropriate. Role=`*` means any
> framework may use the resource. I don't think this is something you would
> want to configure or disable.
> https://mesos.apache.org/documentation/latest/roles/
>
> However i agree with you for the role the scheduler registers as
> (different part of this patch).
>
> Maxim Khutornenko wrote:
> What is the behavior if someone decides to register with a named role,
> say "aurora"? Since we are sorting by RESERVED (which is DEFAULT_ROLE_NAME)
> first, aren't we getting the reverse behavior here? E.g. we'd get anything
> which is NOT "aurora" first (mostly likely only *) and then "aurora"
> resources?
>
> Bill Farner wrote:
> Just to clarify nomenclature, as Zhitao mentioned the naming is off in
> the current draft. Unreserved is `*`.
>
> So the current behavior is to use the unreserved first.
Yes that's the predicate bug I was referring to in my previous reply (still
getting familiar with the collaboration and replying on review board).
This should be fixed in my last update, and a change in test actually picked
the behavior up.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114287
---
On Jan. 13, 2016, 11:50 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 13, 2016, 11:50 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 14, 2016, 1:06 a.m., Dmitriy Shirchenko wrote:
> > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java,
> > lines 216-217
> > <https://reviews.apache.org/r/42126/diff/8/?file=1196146#file1196146line216>
> >
> > Is this intentionally commented out?
>
> Zhitao Li wrote:
> Yes this is intentially removed. Old behavior of changing this `config`
> here resulted in `resources` in static final object `OFFER` not sufficient to
> be allocated, and I don't see any point to use a different executor for this
> test case, so I've deleted this code.
>
> If any reviewer thinks there is a reason to test this with a different
> executor, I can reconstruct an `Offer` object with correct resource and add
> this back.
>
> Bill Farner wrote:
> This is slightly suspicious...are you sure resource accounting wasn't
> impacted in some way? I would expect old test cases to logically work with
> this feature.
I recalled why I commented this out in the first pass now (this was a bit
tricky and I mistakenly thought it's logical bug while it's due to floating
point precision).
In TaskExecutors.java, `SOME_OVERHEAD_EXECUTOR` uses `0.01` cpus:
```
public static final ExecutorSettings SOME_OVERHEAD_EXECUTOR =
TestExecutorSettings.thermosOnlyWithOverhead(
new ResourceSlot(0.01, Amount.of(256L, Data.MB), Amount.of(0L,
Data.MB), 0));
```
Unfortunately, `5.01 - 5 - 0.01` **is not zero** in floating point math of
Java, so the test keeps throwing `InsufficientResourcesException` even if I
constructed an `Offer` whose resource is correctly equivalent to
`ResourceSlot.from(TASK_CONFIG).add(TaskExecutors.SOME_OVERHEAD_EXECUTOR.getExecutorOverhead())`.
**My proposal is to change all comparison with zero in `AcceptedOffer` class (3
places in current patch) to compare with an `EPSILON` whose value is `1e-6` or
something even smaller.** I don't think there is any reasonable usage of
resource unit smaller than this granularity, and the floating point math in the
class would be more robust. We can document this behavior somewhere if it
sounds interesting.
The default resource usage for `THERMOS_EXECUTOR` is 0.25 so I don't encounter
this elsewhere.
Some alternatives which I liked less:
1. Change `TaskExecutors.SOME_OVERHEAD_EXECUTOR`'s cpus to a value which could
be precisely represented by floating point (0.125 or something similar);
2. Manually add a bit more cpu resource to the `Offer` object to trick the test
to pass.
If the above proposed solution sounds fine, I can send an update to the patch.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114353
-----------
On Jan. 14, 2016, 1:39 a.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 14, 2016, 1:39 a.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/ja
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 10:02 p.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Add an entry to the NEWS section for next release.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
NEWS e2b26d9513c15451afb0766fa0aa6f534f6afe49
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 10:32 p.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Rebase and merge changes in NEWS.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114326
---
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java (line 85)
<https://reviews.apache.org/r/42126/#comment175155>
I noticed that this predicate is actually reversed (sorry for the bug).
Fixing in next update.
- Zhitao Li
On Jan. 13, 2016, 10:32 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 13, 2016, 10:32 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 11:50 p.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Fix incorrect `RESERVED` predicate, and all comments from Maxim.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 14, 2016, 12:57 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java,
> > lines 93-94
> > <https://reviews.apache.org/r/42126/diff/8/?file=1196140#file1196140line93>
> >
> > This does not read right to me. If it's empty, a framework will be
> > registered without any role at all.
>
> Bill Farner wrote:
> +1
Will fix to "`The Mesos role this framework will register as. If left empty,
this framework will register without any role and only receive unreserved
resources in offers.`"
> On Jan. 14, 2016, 12:57 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java, line 82
> > <https://reviews.apache.org/r/42126/diff/8/?file=1196137#file1196137line82>
> >
> > Suggest moving all static final fields to the top of this class.
Will do.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114348
-------
On Jan. 13, 2016, 11:50 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 13, 2016, 11:50 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 14, 2016, 1:06 a.m., Dmitriy Shirchenko wrote:
> > src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java,
> > lines 216-217
> > <https://reviews.apache.org/r/42126/diff/8/?file=1196146#file1196146line216>
> >
> > Is this intentionally commented out?
Yes this is intentially removed. Old behavior of changing this `config` here
resulted in `resources` in static final object `OFFER` not sufficient to be
allocated, and I don't see any point to use a different executor for this test
case, so I've deleted this code.
If any reviewer thinks there is a reason to test this with a different
executor, I can reconstruct an `Offer` object with correct resource and add
this back.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114353
---
On Jan. 14, 2016, 1:39 a.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 14, 2016, 1:39 a.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 14, 2016, 12:57 a.m., Maxim Khutornenko wrote:
> > src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java,
> > lines 93-94
> > <https://reviews.apache.org/r/42126/diff/8/?file=1196140#file1196140line93>
> >
> > This does not read right to me. If it's empty, a framework will be
> > registered without any role at all.
>
> Bill Farner wrote:
> +1
>
> Zhitao Li wrote:
> Will fix to "`The Mesos role this framework will register as. If left
> empty, this framework will register without any role and only receive
> unreserved resources in offers.`"
>
> Bill Farner wrote:
> To be clear, I believe Maxim and I are advocating a behavior change - the
> default should be to _not_ set a role (as opposed to role `*`).
I think this is more clear: "`The Mesos role this framework will register as.
The default is to left this empty, and the framework will register without any
role and only receive unreserved resources (role = "*") in offer.`"
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114348
---
On Jan. 13, 2016, 11:50 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 13, 2016, 11:50 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> NEWS acaff9eb2ab184b0ef750f8b8a00c20131997f6b
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
> On Jan. 13, 2016, 2:57 a.m., Bill Farner wrote:
> > src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java, line 129
> > <https://reviews.apache.org/r/42126/diff/3/?file=1195147#file1195147line129>
> >
> > This is a good general-purpose implementation of this behavior, but in
> > practice it seems like overkill. Can you instead match the behavior
> > currently on master and synthesize the `Resource` with
> > `makeMesosRangeResource(PORTS, selectedPorts)`?
I think we still need this function, because ports can be reserved to a
particular role, similar to cpus/mem/disk.
In practice, it seems possible to me that cluster operator decided to partition
available port range among differnet frameworks. Certain Mesos framework
(Cassandra) even requires fixed ports in resource offers to launch their
workload, because their workload only works well with these fixed ports.
Given such possibility, it seems we still need to have correct role for the
ports used when sending them back to Mesos, mostly for handling all possible
configuration purpose.
Please let me know if you think otherwise.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114128
-------
On Jan. 13, 2016, 2:08 a.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 13, 2016, 2:08 a.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new class OfferAllocation, which allcoates resources to
> resources field in TaskInfo and ExecutorInfo from an offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure.
>
>
> Diffs
> -
>
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 5:50 p.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
All review comments from this round, except the one about
`makeMesosRangeResource`, which I replied separately.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42177: Vagrant change to reserve part of the dev cluster's resources to 'aurora-role'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42177/ --- (Updated Jan. 12, 2016, 8:04 p.m.) Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill Farner. Changes --- Added formal review dep to ensure we don't try to commit this before the multi-role support patch. -wfarner Bugs: AURORA-1109 https://issues.apache.org/jira/browse/AURORA-1109 Repository: aurora Description --- This is how I tested changes in https://reviews.apache.org/r/42176/. Diffs - examples/vagrant/upstart/aurora-scheduler.conf 4033184451f36cb5f0233ea96e3dceaae6741275 examples/vagrant/upstart/mesos-master.conf 9d7491c08e14e3951b2fd2f74291a2009883c379 examples/vagrant/upstart/mesos-slave.conf 1ef059b17d16d4f1594a19ff6422ea653a413359 Diff: https://reviews.apache.org/r/42177/diff/ Testing --- Integration test script. Thanks, Zhitao Li
Re: Review Request 42126: New interface to allocate resources of multiple roles from offer.
orTesting` from all methods in
> > this class and make them `private`. It will likely require more
> > fixtures/helpers in the unit test, but the result will be a unit test that
> > is not so highly coupled to the implementation. Do you see anything
> > preventing this approach?
Will do. I think I just need one helper function to be package private so I
don't repeat it in test.
- Zhitao
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review114000
---
On Jan. 12, 2016, 2:29 a.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 12, 2016, 2:29 a.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new interface OfferAllocation (with implementation), which
> allcoates resources to resources field in TaskInfo and ExecutorInfo from an
> offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present. This can be customized with a differnet
> comparator from PREFER_RESERVED in the future when needed.
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure;
> 2. Old code used to construct resources has not been cleaned up yet;
> 3. OfferAllocationImpl's constructor allows to throw, which is a bit awkward.
>
>
> Diffs
> -
>
> src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
>
> src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
> 2255dd407cd1810c7df5baf17cfa85f79bfffeb8
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/AcceptedOfferImplTest.java
> PRE-CREATION
> src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
> e4ae943303823ac4bfbe999ed22f5999484462d8
>
> src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
> 33149ab415292eff04f38b61f2b1d1eac79f347a
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> 1. Unit tested with old and new tests;
> 2. vagrant integration tests: I manually separate out the vagrant box's cpu
> and memory between 'aurora-test' role and '*' and verified that jobs can
> still be launched (I can post the vagrant change in another follow upon
> request).
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 1:34 a.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Better code and test organized by Will's comments, making all lists in the
review immutable.
Also updatd review summyar and description.
Summary (updated)
-
New class to allocate resources of multiple roles from offer.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description (updated)
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: New class to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 13, 2016, 2:08 a.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
Undo incorrect comment find/replace by Intellij.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new class OfferAllocation, which allcoates resources to
resources field in TaskInfo and ExecutorInfo from an offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure.
Diffs (updated)
-
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferTest.java PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Re: Review Request 42126: New interface to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/#review113862
---
src/main/java/org/apache/aurora/scheduler/OfferAllocation.java (line 188)
<https://reviews.apache.org/r/42126/#comment174656>
Hmm, I think I'd like to keep this for two reasons:
1. Even though only one callsite uses value other than `false`, the `false`
value still dictates a `Predicate` to use, so we'll have more code duplicate if
we move this out;
2. Right now aurora only supports `cpus` as revocable resource. I actually
wanted to discuss this separately as current prediction in our company
indicates that we might be more memory bounded. In such a case, we could be
using memory as revocable resources, too.
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java (line 120)
<https://reviews.apache.org/r/42126/#comment174653>
I don't plan on any further usage of the interface other than here in this
series of patches.
The only possible customization that could happen is the reverse ordering
of resource preference (`'*'` vs `reserved role`) but I'd like to withhold it
until someone request such feature.
I think I'll keep the interface definition, change the `AcceptedOfferImpl`
to package private, and use a factory method anyway. We can revisit this later.
How does this sound?
- Zhitao Li
On Jan. 11, 2016, 8:22 p.m., Zhitao Li wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/42126/
> ---
>
> (Updated Jan. 11, 2016, 8:22 p.m.)
>
>
> Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
> Farner.
>
>
> Bugs: AURORA-1109
> https://issues.apache.org/jira/browse/AURORA-1109
>
>
> Repository: aurora
>
>
> Description
> ---
>
> This review is a prototype for introducing multiple role support in Aurora.
> This creates a new interface OfferAllocation (with implementation), which
> allcoates resources to resources field in TaskInfo and ExecutorInfo from an
> offer.
>
> Current implementation prefers reserved resources over shared resources ('*'
> role) if both are present. This can be customized with a differnet
> comparator from PREFER_RESERVED in the future when needed.
>
> Several caveats:
> 1. This performs the allocate after scheduling decision in
> TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency
> and late failure;
> 2. Old code used to construct resources has not been cleaned up yet;
> 3. OfferAllocationImpl's constructor allows to throw, which is a bit awkward.
>
>
> Diffs
> -
>
> src/main/java/org/apache/aurora/scheduler/OfferAllocation.java PRE-CREATION
> src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
> 7c3d681c216b78eeecebbe950186e5a79c6fe982
> src/main/java/org/apache/aurora/scheduler/Resources.java
> db422a959ee7b982c2a44323de41ad75d1a40754
> src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
> 8fdadda67478bb3110aa442b7d78493cf9c3edb4
> src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
> 7e8e456e288986eb0ce92a123b294e1e25d8ed18
> src/test/java/org/apache/aurora/scheduler/OfferAllocationImplTest.java
> PRE-CREATION
>
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
> a5793bffabf4e5d6195b1b99f2363d241c0cecf9
> src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
> 3cbe9acd75def14ae2e0986914ba621fb164b3e4
>
> Diff: https://reviews.apache.org/r/42126/diff/
>
>
> Testing
> ---
>
> Only testing with new and existing unit test. I'll work on more integration
> test once we decide that this is the proper direction.
>
>
> Thanks,
>
> Zhitao Li
>
>
Re: Review Request 42126: New interface to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
(Updated Jan. 12, 2016, 2:29 a.m.)
Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill
Farner.
Changes
---
- Review comments
- Adding 'mesos_role' to command line argument.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new interface OfferAllocation (with implementation), which
allcoates resources to resources field in TaskInfo and ExecutorInfo from an
offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present. This can be customized with a differnet comparator
from PREFER_RESERVED in the future when needed.
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure;
2. Old code used to construct resources has not been cleaned up yet;
3. OfferAllocationImpl's constructor allows to throw, which is a bit awkward.
Diffs (updated)
-
src/main/java/org/apache/aurora/scheduler/AcceptedOffer.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java
2255dd407cd1810c7df5baf17cfa85f79bfffeb8
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/AcceptedOfferImplTest.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/ResourceSlotTest.java
e4ae943303823ac4bfbe999ed22f5999484462d8
src/test/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModuleTest.java
33149ab415292eff04f38b61f2b1d1eac79f347a
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing (updated)
---
1. Unit tested with old and new tests;
2. vagrant integration tests: I manually separate out the vagrant box's cpu and
memory between 'aurora-test' role and '*' and verified that jobs can still be
launched (I can post the vagrant change in another follow upon request).
Thanks,
Zhitao Li
Review Request 42177: Vagrant change to reserve part of the dev cluster's resources to 'aurora-role'
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/42177/ --- Review request for Aurora, Maxim Khutornenko, Dmitriy Shirchenko, and Bill Farner. Bugs: AURORA-1109 https://issues.apache.org/jira/browse/AURORA-1109 Repository: aurora Description --- This is how I tested changes in https://reviews.apache.org/r/42176/. Diffs - examples/vagrant/upstart/aurora-scheduler.conf 4033184451f36cb5f0233ea96e3dceaae6741275 examples/vagrant/upstart/mesos-master.conf 9d7491c08e14e3951b2fd2f74291a2009883c379 examples/vagrant/upstart/mesos-slave.conf 1ef059b17d16d4f1594a19ff6422ea653a413359 Diff: https://reviews.apache.org/r/42177/diff/ Testing --- Integration test script. Thanks, Zhitao Li
Review Request 42125: New interface to allocate resources of multiple roles from offer
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42125/
---
Review request for Aurora, Maxim Khutornenko and Bill Farner.
Summary (updated)
-
New interface to allocate resources of multiple roles from offer
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description (updated)
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new interface OfferAllocation (with implementation), which
allcoates resources to resources field in TaskInfo and ExecutorInfo from an
offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present. This can be customized with a differnet comparator
from PREFER_RESERVED in the future when needed.
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure;
2. Old code used to construct resources has not been cleaned up yet;
3. OfferAllocationImpl's constructor allows to throw, which is a bit awkward.
Diffs
-
Diff: https://reviews.apache.org/r/42125/diff/
Testing (updated)
---
Only testing with new and existing unit test. I'll work on more integration
test once we decide that this is the proper direction.
Thanks,
Zhitao Li
Review Request 42126: New interface to allocate resources of multiple roles from offer.
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/42126/
---
Review request for Aurora, Maxim Khutornenko and Bill Farner.
Bugs: AURORA-1109
https://issues.apache.org/jira/browse/AURORA-1109
Repository: aurora
Description
---
This review is a prototype for introducing multiple role support in Aurora.
This creates a new interface OfferAllocation (with implementation), which
allcoates resources to resources field in TaskInfo and ExecutorInfo from an
offer.
Current implementation prefers reserved resources over shared resources ('*'
role) if both are present. This can be customized with a differnet comparator
from PREFER_RESERVED in the future when needed.
Several caveats:
1. This performs the allocate after scheduling decision in
TaskAssigner.maybeAssign is done, which leaves possibility of inconsistency and
late failure;
2. Old code used to construct resources has not been cleaned up yet;
3. OfferAllocationImpl's constructor allows to throw, which is a bit awkward.
Diffs
-
src/main/java/org/apache/aurora/scheduler/OfferAllocation.java PRE-CREATION
src/main/java/org/apache/aurora/scheduler/ResourceSlot.java
7c3d681c216b78eeecebbe950186e5a79c6fe982
src/main/java/org/apache/aurora/scheduler/Resources.java
db422a959ee7b982c2a44323de41ad75d1a40754
src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java
8fdadda67478bb3110aa442b7d78493cf9c3edb4
src/main/java/org/apache/aurora/scheduler/state/TaskAssigner.java
7e8e456e288986eb0ce92a123b294e1e25d8ed18
src/test/java/org/apache/aurora/scheduler/OfferAllocationImplTest.java
PRE-CREATION
src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java
a5793bffabf4e5d6195b1b99f2363d241c0cecf9
src/test/java/org/apache/aurora/scheduler/state/TaskAssignerImplTest.java
3cbe9acd75def14ae2e0986914ba621fb164b3e4
Diff: https://reviews.apache.org/r/42126/diff/
Testing
---
Only testing with new and existing unit test. I'll work on more integration
test once we decide that this is the proper direction.
Thanks,
Zhitao Li
