date:20160314

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread John Sirois



> On March 14, 2016, 9:51 a.m., Joshua Cohen wrote:
> > Actually, one more thing... should we ensure that the task config it has 
> > either executor config set or uses a Docker container? I think right now 
> > you could configure the scheduler with `-require_docker_use_executor=False` 
> > and send a task with no processes and no Docker container?
> 
> John Sirois wrote:
> The existing code in this change should be enforcing this already; 
> although at a higher layer (the aurora config is checked for processes or 
> Docker container, not the thrift config derived from it (where the executor 
> config is injected)).  Let me know if you see specific misses in the code or 
> unit test coverage.
> 
> John Sirois wrote:
> ... and Bill's code in the scheduler checks the thrift config in this way 
> upon receipt.
> 
> Joshua Cohen wrote:
> This is the scheduler change review where I was asking for this ;).
> 
> I see where we're ensuring that the command line flag is set properly if 
> no executor is configured, but I don't see where we ensure that if the flag 
> is set then the incoming task has either executor config or a docker 
> container? It's possible I'm just missing it though!

Ah sorry - got lost in RBs!


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/#review123427
---


On March 11, 2016, 4:27 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44685/
> ---
> 
> (Updated March 11, 2016, 4:27 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and John Sirois.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is currently labeled as experimental.
> 
> Only the most basic wiring is added here, and assumes that the provided image
> includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
> approach allows containers with an entrypoint, and does not impose environment
> requirements on the image (e.g. python interpreter, libmesos dependencies).
> 
> Note that when using this, other familiar Aurora facilities that relate to the
> thermos executor will not work.  For example, browsing task logs is not
> supported.
> 
> Support for exercising this from the client will come shortly.
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
> 2c56b2eb91a942151865c5a2731ff1cea04b08eb 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
> 2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 60746383fccb107ca27925a91aa1803e2cf0fd85 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> a0d2a717534bbb2e85a556721cc53c1e4b743461 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 1de6966565d2fbd9abd220ad8162b624b109959a 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> a34af4d2fb3863ab8197bcdce942c513d629621b 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
> 708be567ec0f8adbcc53be9553198a8cbb81f3b9 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
>   
> src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java
>  3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  11062e3a097e490c61bfd4dc84990903275521a3 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
> fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  54defc256ad8a261c6b56ee06ad7fdd16a26b057 
>   src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
> ecddc66612751b53b71b917d05d0312aadc16076 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
>  58b4c9312e5c7d12144af56b3896161c039d5288 
>   
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java
>  6a39d89bbba3430961f60a27547c3dfdeb9433f8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> d18ce205412efe08dcd41b6ebc79e29dde9320d1 
>   
>

Re: Review Request 44745: Allow for a pure docker executor.

2016-03-14 Thread Maxim Khutornenko



On March 13, 2016, 12:04 p.m., John Sirois wrote:
> > While your patch is rather easy, I am not sure it is the best way to move 
> > forward. It feels like it is crossing streams with 
> > https://issues.apache.org/jira/browse/AURORA-1288. Putting some thought 
> > into this might be helpful in the long run.
> 
> Bill Farner wrote:
> FWIW i don't think it complicates or even diverges from that ticket.  In 
> my opinion it's yet to be seen whether it's feasible to use the same client 
> for a custom executor (at least, without a decent amount of modularization 
> work).  At the very least, that effort has lost momentum and we shouldn't 
> block progress for it.
> 
> Stephan Erb wrote:
> I mostly brought it up because the ticket also repeatedly mentions the 
> default Mesos command executor. Supporting this one does not sound to 
> different from supporting Docker without Thermos. It would also need similar 
> logic at the UI layer to allow use the Mesos sanbox UI instead of the Thermos 
> one.
> 
> I agree that we should not block progress here. I justed wanted to make 
> sure we are not rushing things (i.e., there isn't even a jira ticket right 
> now).

+1 to Stephan's concerns. The schema changes in this patch don't necessarily 
convey enough meaning to paint a clear picture of where this effort leads us. 
FWICT, nothing in the Task aside from resources is applicable to the Docker 
case and it feels quite hacky to onboard a new executor case this way.

> In my opinion it's yet to be seen whether it's feasible to use the same 
> client for a custom executor (at least, without a decent amount of 
> modularization work).

Bill, would you mind clarifying what this means? Are you expecting this to be a 
purely experimental (POC) effort or is there a solid production quality future 
here? If it's the former, would it be more appropriate to have this effort 
baked in a private branch to avoid possibly unnecessary code churn and review 
cycles?


- Maxim


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44745/#review123314
---


On March 13, 2016, 2:48 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44745/
> ---
> 
> (Updated March 13, 2016, 2:48 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This allows for a job config with no processes defined IFF there is also
> a Docker container defined.  In this case it is assumed that the process
> to run is the Docker container's ENTRYPOINT via the Mesos Docker
> containerizer.
> 
>  src/main/python/apache/aurora/config/__init__.py| 26 
> +++---
>  src/main/python/apache/aurora/config/thrift.py  |  9 +
>  src/test/python/apache/aurora/client/test_config.py | 41 
> +++--
>  src/test/python/apache/aurora/config/test_thrift.py |  5 +
>  4 files changed, 68 insertions(+), 13 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/config/__init__.py 
> 65923be1cb8b88139b8eab0ac5b75428972d3cb1 
>   src/main/python/apache/aurora/config/thrift.py 
> be0cd68674a71bd4baadf276f40a4bc0223ce4be 
>   src/main/python/apache/thermos/config/schema_base.py 
> a6768e67189b0560afef844d6b269bed8ada5f2f 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 88292d3c4423c0555088a0adaee3c0e62ed0567e 
> 
> Diff: https://reviews.apache.org/r/44745/diff/
> 
> 
> Testing
> ---
> 
> Locally green `./build-support/jenkins/build.sh`
> 
> I also patched in https://reviews.apache.org/r/44685/ which this change
> depends on and was able to run scheduler with
> `-allow_docker_parameters -require_docker_use_executor` and successfully
> run this job:
> ```
> import getpass
> 
> jobs=[
>   Service(
> cluster = 'devcluster',
> role = getpass.getuser(),
> environment = 'test',
> name = 'http_example_docker_executor',
> contact = '{{role}}@localhost',
> instances = 1,
> task = Task(
>   name = 'http_docker_example',
>   resources = Resources(cpu=0.4, ram=32*MB, disk=64*MB),
>   processes = []
> ),
> container = Container(
>   docker = Docker(
> image = 'http_example',
> parameters = [
>   Parameter(name = 'env', value = 'HTTP_PORT='),
>   Parameter(name = 'expose', value = ''),
>   Parameter(name = 'publish', value = ':/tcp'),
> ],
>   ),
> ),
>   )
> ]
> ```
> 
> Using the image created with
> `docker build -t http_example

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/#review123414
---


Ship it!




lgtm modulo the below.


src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 (line 187)


This variable/message should probably updated in concert w/ the command 
line parameter rename?


- Joshua Cohen


On March 11, 2016, 11:27 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44685/
> ---
> 
> (Updated March 11, 2016, 11:27 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and John Sirois.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is currently labeled as experimental.
> 
> Only the most basic wiring is added here, and assumes that the provided image
> includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
> approach allows containers with an entrypoint, and does not impose environment
> requirements on the image (e.g. python interpreter, libmesos dependencies).
> 
> Note that when using this, other familiar Aurora facilities that relate to the
> thermos executor will not work.  For example, browsing task logs is not
> supported.
> 
> Support for exercising this from the client will come shortly.
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
> 2c56b2eb91a942151865c5a2731ff1cea04b08eb 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
> 2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 60746383fccb107ca27925a91aa1803e2cf0fd85 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> a0d2a717534bbb2e85a556721cc53c1e4b743461 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 1de6966565d2fbd9abd220ad8162b624b109959a 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> a34af4d2fb3863ab8197bcdce942c513d629621b 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
> 708be567ec0f8adbcc53be9553198a8cbb81f3b9 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
>   
> src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java
>  3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  11062e3a097e490c61bfd4dc84990903275521a3 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
> fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  54defc256ad8a261c6b56ee06ad7fdd16a26b057 
>   src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
> ecddc66612751b53b71b917d05d0312aadc16076 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
>  58b4c9312e5c7d12144af56b3896161c039d5288 
>   
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java
>  6a39d89bbba3430961f60a27547c3dfdeb9433f8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> d18ce205412efe08dcd41b6ebc79e29dde9320d1 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  d3a026cedc86e366d140727dd98265804caed30b 
>   src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
> 860d9607505c0a28c1d0050faa04143b0ab8a097 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> cc8891593c6f31356986f3eb66aabd04ae97acbb 
> 
> Diff: https://reviews.apache.org/r/44685/diff/
> 
> 
> Testing
> ---
> 
> Via additional hacking, i successfully ran the stock [hello 
> world](https://hub.docker.com/_/hello-world/) image.  Within the sandbox, i 
> observed the expected output in the `stdout` file.  Status updates for the 
> task exiting worked as expected.
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread Joshua Cohen



> On March 14, 2016, 3:51 p.m., Joshua Cohen wrote:
> > Actually, one more thing... should we ensure that the task config it has 
> > either executor config set or uses a Docker container? I think right now 
> > you could configure the scheduler with `-require_docker_use_executor=False` 
> > and send a task with no processes and no Docker container?
> 
> John Sirois wrote:
> The existing code in this change should be enforcing this already; 
> although at a higher layer (the aurora config is checked for processes or 
> Docker container, not the thrift config derived from it (where the executor 
> config is injected)).  Let me know if you see specific misses in the code or 
> unit test coverage.
> 
> John Sirois wrote:
> ... and Bill's code in the scheduler checks the thrift config in this way 
> upon receipt.

This is the scheduler change review where I was asking for this ;).

I see where we're ensuring that the command line flag is set properly if no 
executor is configured, but I don't see where we ensure that if the flag is set 
then the incoming task has either executor config or a docker container? It's 
possible I'm just missing it though!


- Joshua


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/#review123427
---


On March 11, 2016, 11:27 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44685/
> ---
> 
> (Updated March 11, 2016, 11:27 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and John Sirois.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is currently labeled as experimental.
> 
> Only the most basic wiring is added here, and assumes that the provided image
> includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
> approach allows containers with an entrypoint, and does not impose environment
> requirements on the image (e.g. python interpreter, libmesos dependencies).
> 
> Note that when using this, other familiar Aurora facilities that relate to the
> thermos executor will not work.  For example, browsing task logs is not
> supported.
> 
> Support for exercising this from the client will come shortly.
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
> 2c56b2eb91a942151865c5a2731ff1cea04b08eb 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
> 2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 60746383fccb107ca27925a91aa1803e2cf0fd85 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> a0d2a717534bbb2e85a556721cc53c1e4b743461 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 1de6966565d2fbd9abd220ad8162b624b109959a 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> a34af4d2fb3863ab8197bcdce942c513d629621b 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
> 708be567ec0f8adbcc53be9553198a8cbb81f3b9 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
>   
> src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java
>  3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  11062e3a097e490c61bfd4dc84990903275521a3 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
> fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  54defc256ad8a261c6b56ee06ad7fdd16a26b057 
>   src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
> ecddc66612751b53b71b917d05d0312aadc16076 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
>  58b4c9312e5c7d12144af56b3896161c039d5288 
>   
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java
>  6a39d89bbba3430961f60a27547c3dfdeb9433f8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> d18ce205412efe08dcd41b6ebc79e29dde9320d1 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>

Re: Review Request 44745: Allow for a pure docker executor.

2016-03-14 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44745/#review123425
---



This seems like something we should cover with the end to end tests? Would you 
mind adding a test that spins up the scheduler to allow executor-less tasks, 
launches a task and then confirms it responds as expected?

Also, if I understand things correctly, the Dockerfile used to build the image 
can differ from our standard http_example image in that we shouldn't require 
libcurl4-nss-dev, since nothing is using mesos.native. Is that right?

- Joshua Cohen


On March 13, 2016, 2:48 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44745/
> ---
> 
> (Updated March 13, 2016, 2:48 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This allows for a job config with no processes defined IFF there is also
> a Docker container defined.  In this case it is assumed that the process
> to run is the Docker container's ENTRYPOINT via the Mesos Docker
> containerizer.
> 
>  src/main/python/apache/aurora/config/__init__.py| 26 
> +++---
>  src/main/python/apache/aurora/config/thrift.py  |  9 +
>  src/test/python/apache/aurora/client/test_config.py | 41 
> +++--
>  src/test/python/apache/aurora/config/test_thrift.py |  5 +
>  4 files changed, 68 insertions(+), 13 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/config/__init__.py 
> 65923be1cb8b88139b8eab0ac5b75428972d3cb1 
>   src/main/python/apache/aurora/config/thrift.py 
> be0cd68674a71bd4baadf276f40a4bc0223ce4be 
>   src/main/python/apache/thermos/config/schema_base.py 
> a6768e67189b0560afef844d6b269bed8ada5f2f 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 88292d3c4423c0555088a0adaee3c0e62ed0567e 
> 
> Diff: https://reviews.apache.org/r/44745/diff/
> 
> 
> Testing
> ---
> 
> Locally green `./build-support/jenkins/build.sh`
> 
> I also patched in https://reviews.apache.org/r/44685/ which this change
> depends on and was able to run scheduler with
> `-allow_docker_parameters -require_docker_use_executor` and successfully
> run this job:
> ```
> import getpass
> 
> jobs=[
>   Service(
> cluster = 'devcluster',
> role = getpass.getuser(),
> environment = 'test',
> name = 'http_example_docker_executor',
> contact = '{{role}}@localhost',
> instances = 1,
> task = Task(
>   name = 'http_docker_example',
>   resources = Resources(cpu=0.4, ram=32*MB, disk=64*MB),
>   processes = []
> ),
> container = Container(
>   docker = Docker(
> image = 'http_example',
> parameters = [
>   Parameter(name = 'env', value = 'HTTP_PORT='),
>   Parameter(name = 'expose', value = ''),
>   Parameter(name = 'publish', value = ':/tcp'),
> ],
>   ),
> ),
>   )
> ]
> ```
> 
> Using the image created with
> `docker build -t http_example src/test/sh/org/apache/aurora/e2e` from:
> ```
> FROM python:2.7
> 
> # mesos.native requires libcurl-nss to initialize MesosExecutorDriver
> RUN apt-get update && apt-get -y install libcurl4-nss-dev
> 
> COPY http_example.py /tmp/
> ENTRYPOINT python /tmp/http_example.py $HTTP_PORT
> ```
> 
> I could connect to http://aurora.local: and get `Hello!` back.
> 
> 
> Thanks,
> 
> John Sirois
> 
>

Re: Review Request 44745: Allow for a pure docker executor.

2016-03-14 Thread John Sirois



> On March 14, 2016, 9:31 a.m., Joshua Cohen wrote:
> > This seems like something we should cover with the end to end tests? Would 
> > you mind adding a test that spins up the scheduler to allow executor-less 
> > tasks, launches a task and then confirms it responds as expected?
> > 
> > Also, if I understand things correctly, the Dockerfile used to build the 
> > image can differ from our standard http_example image in that we shouldn't 
> > require libcurl4-nss-dev, since nothing is using mesos.native. Is that 
> > right?

Sounds good and that's right.  I was holding off additionally on docs to 
explain this alternate mode of operation.  I'll put in that effort as well 
since by your feedback on https://reviews.apache.org/r/44685/ indicates 
willingness to accept this pair of changes.


- John


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44745/#review123425
---


On March 12, 2016, 7:48 p.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44745/
> ---
> 
> (Updated March 12, 2016, 7:48 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This allows for a job config with no processes defined IFF there is also
> a Docker container defined.  In this case it is assumed that the process
> to run is the Docker container's ENTRYPOINT via the Mesos Docker
> containerizer.
> 
>  src/main/python/apache/aurora/config/__init__.py| 26 
> +++---
>  src/main/python/apache/aurora/config/thrift.py  |  9 +
>  src/test/python/apache/aurora/client/test_config.py | 41 
> +++--
>  src/test/python/apache/aurora/config/test_thrift.py |  5 +
>  4 files changed, 68 insertions(+), 13 deletions(-)
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/config/__init__.py 
> 65923be1cb8b88139b8eab0ac5b75428972d3cb1 
>   src/main/python/apache/aurora/config/thrift.py 
> be0cd68674a71bd4baadf276f40a4bc0223ce4be 
>   src/main/python/apache/thermos/config/schema_base.py 
> a6768e67189b0560afef844d6b269bed8ada5f2f 
>   src/test/python/apache/aurora/client/test_config.py 
> b1a3c1865819899ef19173be0f861783a2631d0a 
>   src/test/python/apache/aurora/config/test_thrift.py 
> 88292d3c4423c0555088a0adaee3c0e62ed0567e 
> 
> Diff: https://reviews.apache.org/r/44745/diff/
> 
> 
> Testing
> ---
> 
> Locally green `./build-support/jenkins/build.sh`
> 
> I also patched in https://reviews.apache.org/r/44685/ which this change
> depends on and was able to run scheduler with
> `-allow_docker_parameters -require_docker_use_executor` and successfully
> run this job:
> ```
> import getpass
> 
> jobs=[
>   Service(
> cluster = 'devcluster',
> role = getpass.getuser(),
> environment = 'test',
> name = 'http_example_docker_executor',
> contact = '{{role}}@localhost',
> instances = 1,
> task = Task(
>   name = 'http_docker_example',
>   resources = Resources(cpu=0.4, ram=32*MB, disk=64*MB),
>   processes = []
> ),
> container = Container(
>   docker = Docker(
> image = 'http_example',
> parameters = [
>   Parameter(name = 'env', value = 'HTTP_PORT='),
>   Parameter(name = 'expose', value = ''),
>   Parameter(name = 'publish', value = ':/tcp'),
> ],
>   ),
> ),
>   )
> ]
> ```
> 
> Using the image created with
> `docker build -t http_example src/test/sh/org/apache/aurora/e2e` from:
> ```
> FROM python:2.7
> 
> # mesos.native requires libcurl-nss to initialize MesosExecutorDriver
> RUN apt-get update && apt-get -y install libcurl4-nss-dev
> 
> COPY http_example.py /tmp/
> ENTRYPOINT python /tmp/http_example.py $HTTP_PORT
> ```
> 
> I could connect to http://aurora.local: and get `Hello!` back.
> 
> 
> Thanks,
> 
> John Sirois
> 
>

Review Request 44799: Fixup rpm builder and test docs.

2016-03-14 Thread John Sirois


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44799/
---

Review request for Aurora and Bill Farner.


Repository: aurora-packaging


Description
---

This adjusts to fallout from https://reviews.apache.org/r/43936/ to get
the centos-7 builder working again and improves the test docs to include
both locally built rpm and released (or release candidate) rpm testing.

This change also imporves the `release-candidate` script to be
idempotent and handle missing binaries (ie: when only using a subset of
builders).

 build-support/release/release-candidate | 33 +
 specs/rpm/Makefile  |  5 ++---
 specs/rpm/aurora.spec   |  2 +-
 test/rpm/centos-7/README.md | 24 +---
 4 files changed, 45 insertions(+), 19 deletions(-)


Diffs
-

  build-support/release/release-candidate 
c21a0107043c76137daace8823dd28ac3c46957d 
  specs/rpm/Makefile 0fa215854df9678adf7f9a1209dfca1cc4db8ce9 
  specs/rpm/aurora.spec 20a6d349eb7cd6c801ddc7b149dd656f4a517cd1 
  test/rpm/centos-7/README.md 6a6e419eae0b91608450d932139e515ff47c8445 

Diff: https://reviews.apache.org/r/44799/diff/


Testing
---

CentOS 7 test doc fixes rendered here:
  https://github.com/jsirois/aurora-packaging/tree/0.12.x/test/rpm/centos-7
  
I exercised the `Makefile` and `aurora.spec` fixes to build a new batch of
rpms and then used the improved `release-candidate` script to create the rpm
`upload.tar` and update the 0.12.0 RC bintray repo here:
  https://bintray.com/john-sirois/aurora/centos-7/0.12.0


Thanks,

John Sirois

Re: Review Request 44763: Remove cyrus-sasl and docker RPM dpendencies.

2016-03-14 Thread John Sirois


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44763/#review123461
---



I'm going to patch this in presently to base my rpm tweaks for RC2 on.

- John Sirois


On March 13, 2016, 1:50 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44763/
> ---
> 
> (Updated March 13, 2016, 1:50 p.m.)
> 
> 
> Review request for Aurora, John Sirois and Stephan Erb.
> 
> 
> Repository: aurora-packaging
> 
> 
> Description
> ---
> 
> I don't believe either of these dependencies are valid for the executor.
> 
> 
> Diffs
> -
> 
>   specs/rpm/aurora.spec 291226da27285bf9f7c225ba4fd337691c6e8181 
> 
> Diff: https://reviews.apache.org/r/44763/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread John Sirois


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/#review123496
---


Ship it!




Ship It!

- John Sirois


On March 14, 2016, 1:43 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44685/
> ---
> 
> (Updated March 14, 2016, 1:43 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and John Sirois.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is currently labeled as experimental.
> 
> Only the most basic wiring is added here, and assumes that the provided image
> includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
> approach allows containers with an entrypoint, and does not impose environment
> requirements on the image (e.g. python interpreter, libmesos dependencies).
> 
> Note that when using this, other familiar Aurora facilities that relate to the
> thermos executor will not work.  For example, browsing task logs is not
> supported.
> 
> Support for exercising this from the client will come shortly.
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
> 2c56b2eb91a942151865c5a2731ff1cea04b08eb 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
> 2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 60746383fccb107ca27925a91aa1803e2cf0fd85 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> a0d2a717534bbb2e85a556721cc53c1e4b743461 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 1de6966565d2fbd9abd220ad8162b624b109959a 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> a34af4d2fb3863ab8197bcdce942c513d629621b 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
> 708be567ec0f8adbcc53be9553198a8cbb81f3b9 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
>   
> src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java
>  3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  11062e3a097e490c61bfd4dc84990903275521a3 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
> fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  54defc256ad8a261c6b56ee06ad7fdd16a26b057 
>   src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
> ecddc66612751b53b71b917d05d0312aadc16076 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
>  58b4c9312e5c7d12144af56b3896161c039d5288 
>   
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java
>  6a39d89bbba3430961f60a27547c3dfdeb9433f8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> d18ce205412efe08dcd41b6ebc79e29dde9320d1 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  d3a026cedc86e366d140727dd98265804caed30b 
>   src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
> 860d9607505c0a28c1d0050faa04143b0ab8a097 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> cc8891593c6f31356986f3eb66aabd04ae97acbb 
> 
> Diff: https://reviews.apache.org/r/44685/diff/
> 
> 
> Testing
> ---
> 
> Via additional hacking, i successfully ran the stock [hello 
> world](https://hub.docker.com/_/hello-world/) image.  Within the sandbox, i 
> observed the expected output in the `stdout` file.  Status updates for the 
> task exiting worked as expected.
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Bill Farner


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/
---

Review request for Aurora, Stephan Erb and Zameer Manji.


Repository: aurora


Description
---

Following through with in-flight proposal on the dev list: 
http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E


Diffs
-

  NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 

Diff: https://reviews.apache.org/r/44806/diff/


Testing
---

Rendered at https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md


Thanks,

Bill Farner

Re: Review Request 44532: Allow overriding hostname before announcing

2016-03-14 Thread Bill Farner



> On March 10, 2016, 2:04 p.m., Stephan Erb wrote:
> > src/main/python/apache/aurora/executor/bin/thermos_executor_main.py, line 95
> > 
> >
> > Specifying the command here feels kind of weird. All other options  we 
> > have take the value directly.
> 
> Kunal Thakar wrote:
> Executor arguments are set in stone after starting the scheduler as they 
> are passed to the scheduler through -thermos_executor_flags, so we have to do 
> it this way.
> 
> Stephan Erb wrote:
> Another way to do it would be if you use a wrapper script to launch your 
> Thermos executor. If necessary, you can copy this into the sandbox using the 
> `-thermos_executor_resources` flag. 
> 
> Personally, I would perfer such a solution:
> 
> * It can be used for all options.
> * Thermos does not need to handle the case if the command fails. It is up 
> to the operator writing the wrapper script to handle that case.

+1, a wrapper script superior here as it enables more broad customization 
without the need for piecemeal plumbing from the scheduler.


- Bill


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44532/#review123011
---


On March 8, 2016, 2:33 p.m., Kunal Thakar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44532/
> ---
> 
> (Updated March 8, 2016, 2:33 p.m.)
> 
> 
> Review request for Aurora.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Allow overriding hostname before announcing
> 
> 
> Diffs
> -
> 
>   src/main/python/apache/aurora/executor/bin/thermos_executor_main.py 
> f82858c528808d2a9e77bb56f16e897cfb5bbe73 
>   src/main/python/apache/aurora/executor/common/announcer.py 
> 34e36e0a59093468a8934f58bacb68512949347c 
> 
> Diff: https://reviews.apache.org/r/44532/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Kunal Thakar
> 
>

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread Joshua Cohen


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/#review123504
---


Ship it!




Ship It!

- Joshua Cohen


On March 14, 2016, 7:43 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44685/
> ---
> 
> (Updated March 14, 2016, 7:43 p.m.)
> 
> 
> Review request for Aurora, Joshua Cohen and John Sirois.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> This is currently labeled as experimental.
> 
> Only the most basic wiring is added here, and assumes that the provided image
> includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
> approach allows containers with an entrypoint, and does not impose environment
> requirements on the image (e.g. python interpreter, libmesos dependencies).
> 
> Note that when using this, other familiar Aurora facilities that relate to the
> thermos executor will not work.  For example, browsing task logs is not
> supported.
> 
> Support for exercising this from the client will come shortly.
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
>   src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
> 2c56b2eb91a942151865c5a2731ff1cea04b08eb 
>   src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
> 2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
>   src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
> 60746383fccb107ca27925a91aa1803e2cf0fd85 
>   src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
> a0d2a717534bbb2e85a556721cc53c1e4b743461 
>   src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
> 1de6966565d2fbd9abd220ad8162b624b109959a 
>   
> src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
>  e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
>   src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
> a34af4d2fb3863ab8197bcdce942c513d629621b 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
> ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
>   src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
> 708be567ec0f8adbcc53be9553198a8cbb81f3b9 
>   src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
> c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
>   
> src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java
>  3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
>   
> src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
>  11062e3a097e490c61bfd4dc84990903275521a3 
>   
> src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
> 3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
> fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java
>  54defc256ad8a261c6b56ee06ad7fdd16a26b057 
>   src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
> ecddc66612751b53b71b917d05d0312aadc16076 
>   
> src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
>  58b4c9312e5c7d12144af56b3896161c039d5288 
>   
> src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java
>  6a39d89bbba3430961f60a27547c3dfdeb9433f8 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
> d18ce205412efe08dcd41b6ebc79e29dde9320d1 
>   
> src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java
>  d3a026cedc86e366d140727dd98265804caed30b 
>   src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
> 860d9607505c0a28c1d0050faa04143b0ab8a097 
>   src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
> cc8891593c6f31356986f3eb66aabd04ae97acbb 
> 
> Diff: https://reviews.apache.org/r/44685/diff/
> 
> 
> Testing
> ---
> 
> Via additional hacking, i successfully ran the stock [hello 
> world](https://hub.docker.com/_/hello-world/) image.  Within the sandbox, i 
> observed the expected output in the `stdout` file.  Status updates for the 
> task exiting worked as expected.
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44770: Create scheduler-configuration.md

2016-03-14 Thread Bill Farner


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44770/#review123501
---


Ship it!





docs/scheduler-configuration.md (line 3)


Please wrap this text at 100 cols.  While not historically applied, this is 
the convention.

I'd advise against wrapping the pre-formatted text below, hoever, so you 
can leave it verbatim.


- Bill Farner


On March 13, 2016, 4:21 p.m., Stephan Erb wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44770/
> ---
> 
> (Updated March 13, 2016, 4:21 p.m.)
> 
> 
> Review request for Aurora and Bill Farner.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Having a simple dump of the all scheduler configuration options makes it much 
> easier to discover or find them via a public search engine.
> 
> Ideally we would have a way to update this automatically, but anything is 
> better than not having them listed at all.
> 
> 
> Diffs
> -
> 
>   docs/README.md 78f062abb86faf9ac0347c02828d78a544c74198 
>   docs/deploying-aurora-scheduler.md 03ee360a398821dcc8ca74001bfac478241ad5d8 
>   docs/scheduler-configuration.md PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/44770/diff/
> 
> 
> Testing
> ---
> 
> Rendered version online at 
> https://github.com/StephanErb/aurora/blob/cmdopts/docs/scheduler-configuration.md
> 
> 
> Thanks,
> 
> Stephan Erb
> 
>

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Maxim Khutornenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/#review123507
---


Ship it!





NEWS (line 21)


Missing '.' at the end.


- Maxim Khutornenko


On March 14, 2016, 8:44 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44806/
> ---
> 
> (Updated March 14, 2016, 8:44 p.m.)
> 
> 
> Review request for Aurora, Stephan Erb and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Following through with in-flight proposal on the dev list: 
> http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
> 
> Diff: https://reviews.apache.org/r/44806/diff/
> 
> 
> Testing
> ---
> 
> Rendered at 
> https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44799: Fixup rpm builder and test docs.

2016-03-14 Thread Bill Farner


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44799/#review123482
---


Ship it!




Ship It!

- Bill Farner


On March 14, 2016, 11:53 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44799/
> ---
> 
> (Updated March 14, 2016, 11:53 a.m.)
> 
> 
> Review request for Aurora and Bill Farner.
> 
> 
> Repository: aurora-packaging
> 
> 
> Description
> ---
> 
> This adjusts to fallout from https://reviews.apache.org/r/43936/ to get
> the centos-7 builder working again and improves the test docs to include
> both locally built rpm and released (or release candidate) rpm testing.
> 
> This change also imporves the `release-candidate` script to be
> idempotent and handle missing binaries (ie: when only using a subset of
> builders).
> 
>  build-support/release/release-candidate | 33 
> +
>  specs/rpm/Makefile  |  5 ++---
>  specs/rpm/aurora.spec   |  2 +-
>  test/rpm/centos-7/README.md | 24 +---
>  4 files changed, 45 insertions(+), 19 deletions(-)
> 
> 
> Diffs
> -
> 
>   build-support/release/release-candidate 
> c21a0107043c76137daace8823dd28ac3c46957d 
>   specs/rpm/Makefile 0fa215854df9678adf7f9a1209dfca1cc4db8ce9 
>   specs/rpm/aurora.spec 20a6d349eb7cd6c801ddc7b149dd656f4a517cd1 
>   test/rpm/centos-7/README.md 6a6e419eae0b91608450d932139e515ff47c8445 
> 
> Diff: https://reviews.apache.org/r/44799/diff/
> 
> 
> Testing
> ---
> 
> CentOS 7 test doc fixes rendered here:
>   https://github.com/jsirois/aurora-packaging/tree/0.12.x/test/rpm/centos-7
>   
> I exercised the `Makefile` and `aurora.spec` fixes to build a new batch of
> rpms and then used the improved `release-candidate` script to create the rpm
> `upload.tar` and update the 0.12.0 RC bintray repo here:
>   https://bintray.com/john-sirois/aurora/centos-7/0.12.0
> 
> 
> Thanks,
> 
> John Sirois
> 
>

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

2016-03-14 Thread Bill Farner


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44685/
---

(Updated March 14, 2016, 12:43 p.m.)


Review request for Aurora, Joshua Cohen and John Sirois.


Repository: aurora


Description
---

This is currently labeled as experimental.

Only the most basic wiring is added here, and assumes that the provided image
includes an ENTRYPOINT.  Unlike Docker support via the thermos executor, this
approach allows containers with an entrypoint, and does not impose environment
requirements on the image (e.g. python interpreter, libmesos dependencies).

Note that when using this, other familiar Aurora facilities that relate to the
thermos executor will not work.  For example, browsing task logs is not
supported.

Support for exercising this from the client will come shortly.


Diffs (updated)
-

  NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
  src/jmh/java/org/apache/aurora/benchmark/SnapshotBenchmarks.java 
2c56b2eb91a942151865c5a2731ff1cea04b08eb 
  src/jmh/java/org/apache/aurora/benchmark/TaskStoreBenchmarks.java 
2ec6abd04a454cdcf23f8480c3d14aea85bcb65d 
  src/jmh/java/org/apache/aurora/benchmark/ThriftApiBenchmarks.java 
60746383fccb107ca27925a91aa1803e2cf0fd85 
  src/main/java/org/apache/aurora/scheduler/app/AppModule.java 
a0d2a717534bbb2e85a556721cc53c1e4b743461 
  src/main/java/org/apache/aurora/scheduler/base/TaskTestUtil.java 
1de6966565d2fbd9abd220ad8162b624b109959a 
  
src/main/java/org/apache/aurora/scheduler/configuration/ConfigurationManager.java
 e700fa3550312bfa9c8a3adb25d135f6f500c4b5 
  src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java 
a34af4d2fb3863ab8197bcdce942c513d629621b 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbModule.java 
ff663fa6ea9c5f8b05dabf173ab8ee518fbdd055 
  src/main/java/org/apache/aurora/scheduler/storage/db/DbUtil.java 
708be567ec0f8adbcc53be9553198a8cbb81f3b9 
  src/main/resources/org/apache/aurora/scheduler/storage/db/schema.sql 
c6c1f0a0f1ea64e27dd2b1aaf69ae5cd841c5657 
  
src/test/java/org/apache/aurora/scheduler/app/local/FakeNonVolatileStorage.java 
3336f8cddb2a7a6e8c9e4bc6664708342ab97979 
  
src/test/java/org/apache/aurora/scheduler/configuration/ConfigurationManagerTest.java
 11062e3a097e490c61bfd4dc84990903275521a3 
  src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java 
3db531b52fb2bd94b4b5ce62e6554b5a85ed3ea8 
  src/test/java/org/apache/aurora/scheduler/storage/db/DbCronJobStoreTest.java 
fa0d0dcd9975fd28b92e7a9ccddecbb9a01c01d8 
  
src/test/java/org/apache/aurora/scheduler/storage/db/DbJobUpdateStoreTest.java 
54defc256ad8a261c6b56ee06ad7fdd16a26b057 
  src/test/java/org/apache/aurora/scheduler/storage/db/DbTaskStoreTest.java 
ecddc66612751b53b71b917d05d0312aadc16076 
  
src/test/java/org/apache/aurora/scheduler/storage/db/RowGarbageCollectorTest.java
 58b4c9312e5c7d12144af56b3896161c039d5288 
  
src/test/java/org/apache/aurora/scheduler/storage/log/SnapshotStoreImplIT.java 
6a39d89bbba3430961f60a27547c3dfdeb9433f8 
  src/test/java/org/apache/aurora/scheduler/storage/mem/InMemTaskStoreTest.java 
d18ce205412efe08dcd41b6ebc79e29dde9320d1 
  
src/test/java/org/apache/aurora/scheduler/storage/mem/MemCronJobStoreTest.java 
d3a026cedc86e366d140727dd98265804caed30b 
  src/test/java/org/apache/aurora/scheduler/thrift/ThriftIT.java 
860d9607505c0a28c1d0050faa04143b0ab8a097 
  src/test/java/org/apache/aurora/scheduler/updater/JobUpdaterIT.java 
cc8891593c6f31356986f3eb66aabd04ae97acbb 

Diff: https://reviews.apache.org/r/44685/diff/


Testing
---

Via additional hacking, i successfully ran the stock [hello 
world](https://hub.docker.com/_/hello-world/) image.  Within the sandbox, i 
observed the expected output in the `stdout` file.  Status updates for the task 
exiting worked as expected.


Thanks,

Bill Farner

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Zameer Manji


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/#review123548
---


Ship it!




Ship It!

- Zameer Manji


On March 14, 2016, 1:44 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44806/
> ---
> 
> (Updated March 14, 2016, 1:44 p.m.)
> 
> 
> Review request for Aurora, Stephan Erb and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Following through with in-flight proposal on the dev list: 
> http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
> 
> Diff: https://reviews.apache.org/r/44806/diff/
> 
> 
> Testing
> ---
> 
> Rendered at 
> https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Jake Farrell


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/#review123554
---


Ship it!




Ship It!

- Jake Farrell


On March 14, 2016, 8:44 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44806/
> ---
> 
> (Updated March 14, 2016, 8:44 p.m.)
> 
> 
> Review request for Aurora, Stephan Erb and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Following through with in-flight proposal on the dev list: 
> http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
> 
> Diff: https://reviews.apache.org/r/44806/diff/
> 
> 
> Testing
> ---
> 
> Rendered at 
> https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Stephan Erb


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/#review123542
---


Ship it!




Ship It!

- Stephan Erb


On March 14, 2016, 9:44 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44806/
> ---
> 
> (Updated March 14, 2016, 9:44 p.m.)
> 
> 
> Review request for Aurora, Stephan Erb and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Following through with in-flight proposal on the dev list: 
> http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
> 
> Diff: https://reviews.apache.org/r/44806/diff/
> 
> 
> Testing
> ---
> 
> Rendered at 
> https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44745: Allow for a pure docker executor.

2016-03-14 Thread Stephan Erb



On March 13, 2016, 1:04 p.m., John Sirois wrote:
> > While your patch is rather easy, I am not sure it is the best way to move 
> > forward. It feels like it is crossing streams with 
> > https://issues.apache.org/jira/browse/AURORA-1288. Putting some thought 
> > into this might be helpful in the long run.
> 
> Bill Farner wrote:
> FWIW i don't think it complicates or even diverges from that ticket.  In 
> my opinion it's yet to be seen whether it's feasible to use the same client 
> for a custom executor (at least, without a decent amount of modularization 
> work).  At the very least, that effort has lost momentum and we shouldn't 
> block progress for it.
> 
> Stephan Erb wrote:
> I mostly brought it up because the ticket also repeatedly mentions the 
> default Mesos command executor. Supporting this one does not sound to 
> different from supporting Docker without Thermos. It would also need similar 
> logic at the UI layer to allow use the Mesos sanbox UI instead of the Thermos 
> one.
> 
> I agree that we should not block progress here. I justed wanted to make 
> sure we are not rushing things (i.e., there isn't even a jira ticket right 
> now).
> 
> Maxim Khutornenko wrote:
> +1 to Stephan's concerns. The schema changes in this patch don't 
> necessarily convey enough meaning to paint a clear picture of where this 
> effort leads us. FWICT, nothing in the Task aside from resources is 
> applicable to the Docker case and it feels quite hacky to onboard a new 
> executor case this way.
> 
> > In my opinion it's yet to be seen whether it's feasible to use the same 
> client for a custom executor (at least, without a decent amount of 
> modularization work).
> 
> Bill, would you mind clarifying what this means? Are you expecting this 
> to be a purely experimental (POC) effort or is there a solid production 
> quality future here? If it's the former, would it be more appropriate to have 
> this effort baked in a private branch to avoid possibly unnecessary code 
> churn and review cycles?
> 
> Bill Farner wrote:
> | it feels quite hacky to onboard a new executor case this way
> 
> Suggestions solicited!  Just please don't forget that the intent is to 
> offer real, immediate value - the docker support in Aurora today is quite 
> crippled, and this will address the biggest shortcomings (entrypoints, and 
> zero required deps in images).
> 
> | FWICT, nothing in the Task aside from resources is applicable to the 
> Docker case
> 
> This is a good point.  Perhaps we should create a separate struct and 
> field in `Job` for this case?
> 
> | Bill, would you mind clarifying what this means?
> 
> What i mean is that the client, DSL, and executor all have relatively 
> high coupling.  Adding custom executor support in the client will require 
> non-trivial effort to break that coupling.  I would like to avoid blocking 
> this feature on that goal.
> 
> | Are you expecting this to be a purely experimental (POC) effort or is 
> there a solid production quality future here?
> 
> That is very much dependent on the underlying support in mesos.  Today, i 
> see it as the best support for docker containers in mesos.  It's been 
> available for some time, and the work here is entirely plumbing to enable it 
> in Aurora.
> 
> | would it be more appropriate to have this effort baked in a private 
> branch to avoid possibly unnecessary code churn and review cycles?
> 
> I don't foresee enough churn to warrant that.
> 
> John Sirois wrote:
> Noting that I'm backing off this change until sentiment settles one way 
> or the other.  If it settles in-favor I'll address both Stephan & Joshua's 
> feedback at that point.

I have to backoff out of the discussion here, as I don't have the necessary 
cycles to participate. A couple of closing notes from my side:

* I agree with Maxim that giving an empty process list a special meaning feels 
kind of like a hack.
* I probably wouldn't have complained about this if it was that way from the 
beginning...
* Docker support is still considered experimental, so no decision is cast in 
stone. We can change stuff without to much hassle.
* It is great that you are improving the current docker support (even though I 
am a fanboy of the upcoming unified container support :-)


- Stephan


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44745/#review123314
---


On March 13, 2016, 3:48 a.m., John Sirois wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44745/
> ---
> 
> (Updated March 13, 2016, 3:48 a.m.)
> 
> 
> Review request for Aurora, Joshua Cohen

Re: Review Request 44827: Do not split the shell command string passed into shell health check script

2016-03-14 Thread Dmitriy Shirchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44827/
---

(Updated March 15, 2016, 3:49 a.m.)


Review request for Aurora, John Sirois, Bill Farner, and Zameer Manji.


Bugs: AURORA-1633
https://issues.apache.org/jira/browse/AURORA-1633


Repository: aurora


Description
---

Fixing bug where you could not pass in shell command into health checker with 
environment variables. When environment variable assignement was passed in, 
only that part would get executed and 0 would always get returned.


Diffs
-

  src/main/python/apache/aurora/common/health_check/shell.py 
bf63d936b3044dfa97a787938b643a52497f2d79 
  src/test/python/apache/aurora/common/health_check/test_shell.py 
8d3a3e4e259e1ff699854aeb2434ac21f38e49ea 

Diff: https://reviews.apache.org/r/44827/diff/


Testing
---

Unit tests + end_to_end test.


Thanks,

Dmitriy Shirchenko

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

2016-03-14 Thread Aurora ReviewBot


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/44806/#review123518
---


Ship it!




Master (3806e62) is green with this patch.
  ./build-support/jenkins/build.sh

I will refresh this build result if you post a review containing "@ReviewBot 
retry"

- Aurora ReviewBot


On March 14, 2016, 8:44 p.m., Bill Farner wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/44806/
> ---
> 
> (Updated March 14, 2016, 8:44 p.m.)
> 
> 
> Review request for Aurora, Stephan Erb and Zameer Manji.
> 
> 
> Repository: aurora
> 
> 
> Description
> ---
> 
> Following through with in-flight proposal on the dev list: 
> http://mail-archives.apache.org/mod_mbox/aurora-dev/201603.mbox/%3CCAFWq12W%2BM4t31SYTR3SdU2MYdp0ndTuGU1H4RvMhF5D_HFourg%40mail.gmail.com%3E
> 
> 
> Diffs
> -
> 
>   NEWS da3e4cea8ca688b6b7c5bafae67133df065d9255 
> 
> Diff: https://reviews.apache.org/r/44806/diff/
> 
> 
> Testing
> ---
> 
> Rendered at 
> https://github.com/wfarner/aurora/blob/wfarner/news/RELEASE-NOTES.md
> 
> 
> Thanks,
> 
> Bill Farner
> 
>

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Re: Review Request 44745: Allow for a pure docker executor.

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Re: Review Request 44745: Allow for a pure docker executor.

Re: Review Request 44745: Allow for a pure docker executor.

Review Request 44799: Fixup rpm builder and test docs.

Re: Review Request 44763: Remove cyrus-sasl and docker RPM dpendencies.

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Review Request 44806: Rename NEWS to RELEASE-NOTES.md

Re: Review Request 44532: Allow overriding hostname before announcing

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Re: Review Request 44770: Create scheduler-configuration.md

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

Re: Review Request 44799: Fixup rpm builder and test docs.

Re: Review Request 44685: Add scheduler support for running tasks using the mesos Docker containerizer.

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

Re: Review Request 44745: Allow for a pure docker executor.

Re: Review Request 44827: Do not split the shell command string passed into shell health check script

Re: Review Request 44806: Rename NEWS to RELEASE-NOTES.md

22 matches

Site Navigation

Mail list logo

Footer information