Repository: aurora Updated Branches: refs/heads/master 2f480c7f3 -> 915459dac
Set DiscoveryInfo in mesos tasks. This allows alternative service discovery methodologies to find tasks from Aurora (e.g. mesos-dns), especially the dynamic port mapping. Bugs closed: AURORA-1629 Reviewed at https://reviews.apache.org/r/45177/ Project: http://git-wip-us.apache.org/repos/asf/aurora/repo Commit: http://git-wip-us.apache.org/repos/asf/aurora/commit/915459da Tree: http://git-wip-us.apache.org/repos/asf/aurora/tree/915459da Diff: http://git-wip-us.apache.org/repos/asf/aurora/diff/915459da Branch: refs/heads/master Commit: 915459dac76ed0732addce87420a4ba51d916de8 Parents: 2f480c7 Author: Zhitao Li <[email protected]> Authored: Wed Apr 6 21:31:48 2016 -0700 Committer: Bill Farner <[email protected]> Committed: Wed Apr 6 21:31:48 2016 -0700 ---------------------------------------------------------------------- RELEASE-NOTES.md | 3 + docs/features/service-discovery.md | 30 ++++++ docs/reference/scheduler-configuration.md | 3 + examples/vagrant/upstart/aurora-scheduler.conf | 1 + .../configuration/executor/ExecutorModule.java | 10 +- .../executor/ExecutorSettings.java | 8 +- .../scheduler/mesos/MesosTaskFactory.java | 41 +++++++- .../scheduler/mesos/TestExecutorSettings.java | 6 +- .../mesos/MesosTaskFactoryImplTest.java | 103 ++++++++++++++++--- .../apache/aurora/e2e/http/http_example.aurora | 13 ++- .../sh/org/apache/aurora/e2e/test_end_to_end.sh | 44 ++++++++ 11 files changed, 238 insertions(+), 24 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/RELEASE-NOTES.md ---------------------------------------------------------------------- diff --git a/RELEASE-NOTES.md b/RELEASE-NOTES.md index 46fa2d4..ebc252f 100644 --- a/RELEASE-NOTES.md +++ b/RELEASE-NOTES.md @@ -26,6 +26,9 @@ - Added scheduler argument `-require_docker_use_executor` that indicates whether the scheduler should accept tasks that use the Docker containerizer without an executor (experimental). - Jobs referencing invalid tier name will be rejected by the scheduler. +- Added a new scheduler argument `--populate_discovery_info`. If set to true, Aurora will start + to populate DiscoveryInfo field on TaskInfo of Mesos. This could be used for alternative + service discovery solution like Mesos-DNS. ### Deprecations and removals: http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/docs/features/service-discovery.md ---------------------------------------------------------------------- diff --git a/docs/features/service-discovery.md b/docs/features/service-discovery.md index 858ca2a..f242730 100644 --- a/docs/features/service-discovery.md +++ b/docs/features/service-discovery.md @@ -12,3 +12,33 @@ of which there are several reference implementations: These can also be used natively in Finagle using the [ZookeeperServerSetCluster](https://github.com/twitter/finagle/blob/master/finagle-serversets/src/main/scala/com/twitter/finagle/zookeeper/ZookeeperServerSetCluster.scala). For more information about how to configure announcing, see the [Configuration Reference](../reference/configuration.md). + +Using Mesos DiscoveryInfo +------------------------- +Experimental support for populating DiscoveryInfo in Mesos is introduced in Aurora. This can be used to build +custom service discovery system not using zookeeper. Please see `Service Discovery` section in +[Mesos Framework Development guide](http://mesos.apache.org/documentation/latest/app-framework-development-guide/) for +explanation of the protobuf message in Mesos. + +To use this feature, please enable `--populate_discovery_info` flag on scheduler. All jobs started by scheduler +afterwards will have their portmap populated to Mesos and discoverable in `/state` endpoint in Mesos master and agent. + +### Using Mesos DNS +An example is using [Mesos-DNS](https://github.com/mesosphere/mesos-dns), which is able to generate multiple DNS +records. With current implementation, the example job with key `devcluster/vagrant/test/http-example` generates at +least the following: + +1. An A record for `http_example.test.vagrant.twitterscheduler.mesos` (which only includes IP address); +2. A [SRV record](https://en.wikipedia.org/wiki/SRV_record) for + `_http_example.test.vagrant._tcp.twitterscheduler.mesos`, which includes IP address and every port. This should only + be used if the service has one port. +3. A SRV record `_{port-name}._http_example.test.vagrant._tcp.twitterscheduler.mesos` for each port name + defined. This should be used when the service has multiple ports. + +Things to note: + +1. The domain part (".mesos" in above example) can be configured in [Mesos DNS](http://mesosphere.github.io/mesos-dns/docs/configuration-parameters.html); +2. The `twitterscheduler` part is the lower-case of framework name, which is not configurable right now (see + [TWITTER_SCHEDULER_NAME](https://github.com/apache/aurora/blob/master/src/main/java/org/apache/aurora/scheduler/mesos/CommandLineDriverSettingsModule.java#L98)); +3. Right now, portmap and port aliases in announcer object are not reflected in DiscoveryInfo, therefore not visible in + Mesos DNS records either. This is because they are only resolved in thermos executors. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/docs/reference/scheduler-configuration.md ---------------------------------------------------------------------- diff --git a/docs/reference/scheduler-configuration.md b/docs/reference/scheduler-configuration.md index e6c0bb6..f08603a 100644 --- a/docs/reference/scheduler-configuration.md +++ b/docs/reference/scheduler-configuration.md @@ -230,6 +230,9 @@ Optional flags: -offer_reservation_duration=(3, mins) Time to reserve a slave's offers while trying to satisfy a task preempting another. (org.apache.aurora.scheduler.scheduling.SchedulingModule.offer_reservation_duration) +-populate_discovery_info=false + If true, Aurora populates DiscoveryInfo field of Mesos TaskInfo. + (org.apache.aurora.scheduler.configuration.executor.ExecutorModule.populate_discovery_info) -preemption_delay=(3, mins) Time interval after which a pending task becomes eligible to preempt other tasks (org.apache.aurora.scheduler.preemptor.PreemptorModule.preemption_delay) http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/examples/vagrant/upstart/aurora-scheduler.conf ---------------------------------------------------------------------- diff --git a/examples/vagrant/upstart/aurora-scheduler.conf b/examples/vagrant/upstart/aurora-scheduler.conf index d61801c..b9732d2 100644 --- a/examples/vagrant/upstart/aurora-scheduler.conf +++ b/examples/vagrant/upstart/aurora-scheduler.conf @@ -49,4 +49,5 @@ exec bin/aurora-scheduler \ -enable_h2_console=true \ -tier_config=/home/vagrant/aurora/src/main/resources/org/apache/aurora/scheduler/tiers.json \ -mesos_role=aurora-role \ + -populate_discovery_info=true \ -receive_revocable_resources=true http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java ---------------------------------------------------------------------- diff --git a/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java b/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java index add1270..1fe27a5 100644 --- a/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java +++ b/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorModule.java @@ -103,6 +103,10 @@ public class ExecutorModule extends AbstractModule { + "into all (non-mesos) containers.") private static final Arg<List<Volume>> GLOBAL_CONTAINER_MOUNTS = Arg.create(ImmutableList.of()); + @CmdLine(name = "populate_discovery_info", + help = "If true, Aurora populates DiscoveryInfo field of Mesos TaskInfo.") + private static final Arg<Boolean> POPULATE_DISCOVERY_INFO = Arg.create(false); + @VisibleForTesting static CommandInfo makeExecutorCommand( String thermosExecutorPath, @@ -165,7 +169,8 @@ public class ExecutorModule extends AbstractModule { .addResources(makeResource(CPUS, EXECUTOR_OVERHEAD_CPUS.get())) .addResources(makeResource(RAM_MB, EXECUTOR_OVERHEAD_RAM.get().as(Data.MB))) .build(), - volumeMounts)); + volumeMounts), + POPULATE_DISCOVERY_INFO.get()); } private static ExecutorSettings makeCustomExecutorSettings() { @@ -175,7 +180,8 @@ public class ExecutorModule extends AbstractModule { ExecutorSettingsLoader.read( Files.newBufferedReader( CUSTOM_EXECUTOR_CONFIG.get().toPath(), - StandardCharsets.UTF_8))); + StandardCharsets.UTF_8)), + POPULATE_DISCOVERY_INFO.get()); } catch (ExecutorSettingsLoader.ExecutorConfigException | IOException e) { throw new IllegalArgumentException("Failed to read executor settings: " + e, e); } http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java ---------------------------------------------------------------------- diff --git a/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java b/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java index 7beea81..e4279b1 100644 --- a/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java +++ b/src/main/java/org/apache/aurora/scheduler/configuration/executor/ExecutorSettings.java @@ -27,9 +27,11 @@ import static java.util.Objects.requireNonNull; */ public class ExecutorSettings { private final ExecutorConfig config; + private final boolean populateDiscoveryInfo; - public ExecutorSettings(ExecutorConfig config) { + public ExecutorSettings(ExecutorConfig config, boolean populateDiscoveryInfo) { this.config = requireNonNull(config); + this.populateDiscoveryInfo = populateDiscoveryInfo; } public ExecutorConfig getExecutorConfig() { @@ -38,6 +40,10 @@ public class ExecutorSettings { return config; } + public boolean shouldPopulateDiscoverInfo() { + return populateDiscoveryInfo; + } + private double getExecutorResourceValue(ResourceType resource) { return config.getExecutor().getResourcesList().stream() .filter(r -> r.getName().equals(resource.getName())) http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java ---------------------------------------------------------------------- diff --git a/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java b/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java index 85c550b..fb7c7b2 100644 --- a/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java +++ b/src/main/java/org/apache/aurora/scheduler/mesos/MesosTaskFactory.java @@ -14,6 +14,7 @@ package org.apache.aurora.scheduler.mesos; import java.util.List; +import java.util.Map; import java.util.Set; import javax.inject.Inject; @@ -38,15 +39,18 @@ import org.apache.aurora.scheduler.storage.entities.IAssignedTask; import org.apache.aurora.scheduler.storage.entities.IDockerContainer; import org.apache.aurora.scheduler.storage.entities.IJobKey; import org.apache.aurora.scheduler.storage.entities.IMetadata; +import org.apache.aurora.scheduler.storage.entities.IServerInfo; import org.apache.aurora.scheduler.storage.entities.ITaskConfig; import org.apache.mesos.Protos; import org.apache.mesos.Protos.CommandInfo; import org.apache.mesos.Protos.ContainerInfo; +import org.apache.mesos.Protos.DiscoveryInfo; import org.apache.mesos.Protos.ExecutorID; import org.apache.mesos.Protos.ExecutorInfo; import org.apache.mesos.Protos.Label; import org.apache.mesos.Protos.Labels; import org.apache.mesos.Protos.Offer; +import org.apache.mesos.Protos.Port; import org.apache.mesos.Protos.Resource; import org.apache.mesos.Protos.TaskID; import org.apache.mesos.Protos.TaskInfo; @@ -78,13 +82,22 @@ public interface MesosTaskFactory { @VisibleForTesting static final String METADATA_LABEL_PREFIX = "org.apache.aurora.metadata."; + @VisibleForTesting + static final String DEFAULT_PORT_PROTOCOL = "TCP"; + private final ExecutorSettings executorSettings; private final TierManager tierManager; + private final IServerInfo serverInfo; @Inject - MesosTaskFactoryImpl(ExecutorSettings executorSettings, TierManager tierManager) { + MesosTaskFactoryImpl( + ExecutorSettings executorSettings, + TierManager tierManager, + IServerInfo serverInfo) { + this.executorSettings = requireNonNull(executorSettings); this.tierManager = requireNonNull(tierManager); + this.serverInfo = requireNonNull(serverInfo); } @VisibleForTesting @@ -105,6 +118,11 @@ public interface MesosTaskFactory { return String.format("%s.%s", getJobSourceName(task), instanceId); } + @VisibleForTesting + static String getInverseJobSourceName(IJobKey job) { + return String.format("%s.%s.%s", job.getName(), job.getEnvironment(), job.getRole()); + } + private static byte[] serializeTask(IAssignedTask task) throws SchedulerException { try { return ThriftBinaryCodec.encode(task.newBuilder()); @@ -146,6 +164,10 @@ public interface MesosTaskFactory { configureTaskLabels(config.getMetadata(), taskBuilder); + if (executorSettings.shouldPopulateDiscoverInfo()) { + configureDiscoveryInfos(task, taskBuilder); + } + if (config.getContainer().isSetMesos()) { configureTaskForNoContainer(task, taskBuilder, acceptedOffer); } else if (config.getContainer().isSetDocker()) { @@ -220,5 +242,22 @@ public interface MesosTaskFactory { taskBuilder.setLabels(Labels.newBuilder().addAllLabels(labels)); } } + + private void configureDiscoveryInfos(IAssignedTask task, TaskInfo.Builder taskBuilder) { + DiscoveryInfo.Builder builder = taskBuilder.getDiscoveryBuilder(); + builder.setVisibility(DiscoveryInfo.Visibility.CLUSTER); + builder.setName(getInverseJobSourceName(task.getTask().getJob())); + builder.setEnvironment(task.getTask().getJob().getEnvironment()); + // A good sane choice for default location is current Aurora cluster name. + builder.setLocation(serverInfo.getClusterName()); + for (Map.Entry<String, Integer> entry : task.getAssignedPorts().entrySet()) { + builder.getPortsBuilder().addPorts( + Port.newBuilder() + .setName(entry.getKey()) + .setNumber(entry.getValue()) + .setProtocol(DEFAULT_PORT_PROTOCOL) + ); + } + } } } http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java ---------------------------------------------------------------------- diff --git a/src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java b/src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java index 7110fbd..8cef410 100644 --- a/src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java +++ b/src/main/java/org/apache/aurora/scheduler/mesos/TestExecutorSettings.java @@ -66,12 +66,14 @@ public final class TestExecutorSettings { public static final ExecutorConfig THERMOS_CONFIG = new ExecutorConfig(THERMOS_EXECUTOR_INFO, ImmutableList.of()); - public static final ExecutorSettings THERMOS_EXECUTOR = new ExecutorSettings(THERMOS_CONFIG); + public static final ExecutorSettings THERMOS_EXECUTOR = new ExecutorSettings( + THERMOS_CONFIG, false); public static ExecutorSettings thermosOnlyWithOverhead(ResourceSlot overhead) { ExecutorConfig config = THERMOS_EXECUTOR.getExecutorConfig(); ExecutorInfo.Builder executor = config.getExecutor().toBuilder(); executor.clearResources().addAllResources(overhead.toResourceList(TaskTestUtil.DEV_TIER)); - return new ExecutorSettings(new ExecutorConfig(executor.build(), config.getVolumeMounts())); + return new ExecutorSettings( + new ExecutorConfig(executor.build(), config.getVolumeMounts()), false); } } http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java ---------------------------------------------------------------------- diff --git a/src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java b/src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java index 3a60486..4f5ac15 100644 --- a/src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java +++ b/src/test/java/org/apache/aurora/scheduler/mesos/MesosTaskFactoryImplTest.java @@ -13,6 +13,7 @@ */ package org.apache.aurora.scheduler.mesos; +import java.util.Map; import java.util.stream.Collectors; import com.google.common.collect.ImmutableList; @@ -26,6 +27,7 @@ import org.apache.aurora.gen.Container; import org.apache.aurora.gen.DockerContainer; import org.apache.aurora.gen.DockerParameter; import org.apache.aurora.gen.MesosContainer; +import org.apache.aurora.gen.ServerInfo; import org.apache.aurora.gen.TaskConfig; import org.apache.aurora.scheduler.ResourceSlot; import org.apache.aurora.scheduler.ResourceType; @@ -36,6 +38,8 @@ import org.apache.aurora.scheduler.configuration.executor.ExecutorConfig; import org.apache.aurora.scheduler.configuration.executor.ExecutorSettings; import org.apache.aurora.scheduler.mesos.MesosTaskFactory.MesosTaskFactoryImpl; import org.apache.aurora.scheduler.storage.entities.IAssignedTask; +import org.apache.aurora.scheduler.storage.entities.IJobKey; +import org.apache.aurora.scheduler.storage.entities.IServerInfo; import org.apache.aurora.scheduler.storage.entities.ITaskConfig; import org.apache.mesos.Protos; import org.apache.mesos.Protos.ContainerInfo; @@ -55,7 +59,9 @@ import org.junit.Test; import static org.apache.aurora.scheduler.ResourceSlot.makeMesosRangeResource; import static org.apache.aurora.scheduler.base.TaskTestUtil.DEV_TIER; import static org.apache.aurora.scheduler.base.TaskTestUtil.REVOCABLE_TIER; +import static org.apache.aurora.scheduler.mesos.MesosTaskFactory.MesosTaskFactoryImpl.DEFAULT_PORT_PROTOCOL; import static org.apache.aurora.scheduler.mesos.MesosTaskFactory.MesosTaskFactoryImpl.METADATA_LABEL_PREFIX; +import static org.apache.aurora.scheduler.mesos.MesosTaskFactory.MesosTaskFactoryImpl.getInverseJobSourceName; import static org.apache.aurora.scheduler.mesos.TaskExecutors.NO_OVERHEAD_EXECUTOR; import static org.apache.aurora.scheduler.mesos.TaskExecutors.SOME_OVERHEAD_EXECUTOR; import static org.apache.aurora.scheduler.mesos.TestExecutorSettings.THERMOS_CONFIG; @@ -107,6 +113,10 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { .addResources(makeMesosRangeResource(ResourceType.PORTS, ImmutableSet.of(80))) .build(); + private static final String CLUSTER_NAME = "cluster_name"; + private static final IServerInfo SERVER_INFO = IServerInfo.build( + new ServerInfo(CLUSTER_NAME, "")); + private MesosTaskFactory taskFactory; private ExecutorSettings config; private TierManager tierManager; @@ -142,7 +152,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { @Test public void testExecutorInfoUnchanged() { expect(tierManager.getTier(TASK_CONFIG)).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); @@ -150,12 +160,13 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { assertEquals(populateDynamicFields(DEFAULT_EXECUTOR, TASK), task.getExecutor()); checkTaskResources(TASK.getTask(), task); + checkDiscoveryInfoUnset(task); } @Test public void testTaskInfoRevocable() { expect(tierManager.getTier(TASK_CONFIG)).andReturn(REVOCABLE_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); Resource revocableCPU = OFFER_THERMOS_EXECUTOR.getResources(0).toBuilder() .setRevocable(Resource.RevocableInfo.getDefaultInstance()) @@ -170,6 +181,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { TaskInfo task = taskFactory.createFrom(TASK, withRevocable); checkTaskResources(TASK.getTask(), task); assertTrue(task.getResourcesList().stream().anyMatch(Resource::hasRevocable)); + checkDiscoveryInfoUnset(task); } @Test @@ -179,12 +191,13 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { builder.unsetAssignedPorts(); IAssignedTask assignedTask = IAssignedTask.build(builder); expect(tierManager.getTier(assignedTask.getTask())).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); TaskInfo task = taskFactory.createFrom(IAssignedTask.build(builder), OFFER_THERMOS_EXECUTOR); checkTaskResources(ITaskConfig.build(builder.getTask()), task); + checkDiscoveryInfoUnset(task); } @Test @@ -193,7 +206,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { // + executor overhead. We need to ensure we allocate a non-zero amount of ram in this case. config = NO_OVERHEAD_EXECUTOR; expect(tierManager.getTier(TASK_CONFIG)).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); @@ -206,6 +219,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { // Simulate the upsizing needed for the task to meet the minimum thermos requirements. TaskConfig dummyTask = TASK.getTask().newBuilder(); checkTaskResources(ITaskConfig.build(dummyTask), task); + checkDiscoveryInfoUnset(task); } private void checkTaskResources(ITaskConfig task, TaskInfo taskInfo) { @@ -214,6 +228,32 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { getTotalTaskResources(taskInfo)); } + private void checkDiscoveryInfoUnset(TaskInfo taskInfo) { + assertFalse(taskInfo.hasDiscovery()); + } + + private void checkDiscoveryInfo( + TaskInfo taskInfo, + Map<String, Integer> assignedPorts, + IJobKey job) { + + assertTrue(taskInfo.hasDiscovery()); + Protos.DiscoveryInfo.Builder expectedDiscoveryInfo = Protos.DiscoveryInfo.newBuilder() + .setVisibility(Protos.DiscoveryInfo.Visibility.CLUSTER) + .setLocation(CLUSTER_NAME) + .setEnvironment(job.getEnvironment()) + .setName(getInverseJobSourceName(job)); + for (Map.Entry<String, Integer> entry : assignedPorts.entrySet()) { + expectedDiscoveryInfo.getPortsBuilder().addPorts( + Protos.Port.newBuilder() + .setName(entry.getKey()) + .setProtocol(DEFAULT_PORT_PROTOCOL) + .setNumber(entry.getValue())); + } + + assertEquals(expectedDiscoveryInfo.build(), taskInfo.getDiscovery()); + } + private TaskInfo getDockerTaskInfo() { return getDockerTaskInfo(TASK_WITH_DOCKER); } @@ -222,7 +262,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { config = SOME_OVERHEAD_EXECUTOR; expect(tierManager.getTier(task.getTask())).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); @@ -246,17 +286,19 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { @Test public void testGlobalMounts() { - config = new ExecutorSettings(new ExecutorConfig( - TestExecutorSettings.THERMOS_EXECUTOR_INFO, - ImmutableList.of( - Volume.newBuilder() - .setHostPath("/host") - .setContainerPath("/container") - .setMode(Mode.RO) - .build()))); + config = new ExecutorSettings( + new ExecutorConfig( + TestExecutorSettings.THERMOS_EXECUTOR_INFO, + ImmutableList.of( + Volume.newBuilder() + .setHostPath("/host") + .setContainerPath("/container") + .setMode(Mode.RO) + .build())), + false); expect(tierManager.getTier(TASK_WITH_DOCKER.getTask())).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); @@ -269,7 +311,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { @Test public void testMetadataLabelMapping() { expect(tierManager.getTier(TASK.getTask())).andReturn(DEV_TIER); - taskFactory = new MesosTaskFactoryImpl(config, tierManager); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); control.replay(); @@ -283,6 +325,7 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { .collect(GuavaUtils.toImmutableSet()); assertEquals(labels, metadata); + checkDiscoveryInfoUnset(task); } @Test @@ -303,6 +346,36 @@ public class MesosTaskFactoryImplTest extends EasyMockTest { .setImage("hello-world")) .build(); assertEquals(expectedContainer, task.getContainer()); + checkDiscoveryInfoUnset(task); + } + + @Test + public void testPopulateDiscoveryInfoNoPort() { + config = new ExecutorSettings(THERMOS_CONFIG, true); + AssignedTask builder = TASK.newBuilder(); + builder.getTask().unsetRequestedPorts(); + builder.unsetAssignedPorts(); + IAssignedTask assignedTask = IAssignedTask.build(builder); + expect(tierManager.getTier(assignedTask.getTask())).andReturn(DEV_TIER); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); + + control.replay(); + + TaskInfo task = taskFactory.createFrom(IAssignedTask.build(builder), OFFER_THERMOS_EXECUTOR); + checkTaskResources(ITaskConfig.build(builder.getTask()), task); + checkDiscoveryInfo(task, ImmutableMap.of(), assignedTask.getTask().getJob()); + } + + @Test + public void testPopulateDiscoveryInfo() { + config = new ExecutorSettings(THERMOS_CONFIG, true); + expect(tierManager.getTier(TASK_CONFIG)).andReturn(DEV_TIER); + taskFactory = new MesosTaskFactoryImpl(config, tierManager, SERVER_INFO); + + control.replay(); + TaskInfo task = taskFactory.createFrom(TASK, OFFER_THERMOS_EXECUTOR); + checkTaskResources(TASK.getTask(), task); + checkDiscoveryInfo(task, ImmutableMap.of("http", 80), TASK.getTask().getJob()); } private static ResourceSlot getTotalTaskResources(TaskInfo task) { http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora ---------------------------------------------------------------------- diff --git a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora index bb4fdec..2813b6c 100644 --- a/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora +++ b/src/test/sh/org/apache/aurora/e2e/http/http_example.aurora @@ -16,6 +16,11 @@ import getpass DEFAULT_CMD = 'cp /vagrant/src/test/sh/org/apache/aurora/e2e/http_example.py .' +echo_ports = Process( + name = 'echo_ports', + cmdline = 'echo "tcp port: {{thermos.ports[tcp]}}; http port: {{thermos.ports[http]}}; alias: {{thermos.ports[alias]}}"' +) + run_server = Process( name = 'run_server', cmdline = 'python http_example.py {{thermos.ports[http]}}') @@ -28,8 +33,8 @@ stage_server = Process( test_task = Task( name = 'http_example', resources = Resources(cpu=0.4, ram=32*MB, disk=64*MB), - processes = [stage_server, run_server], - constraints = order(stage_server, run_server)) + processes = [echo_ports, stage_server, run_server], + constraints = order(echo_ports, stage_server, run_server)) update_config = UpdateConfig(watch_secs=10, batch_size=2) health_check_config = HealthCheckConfig(initial_interval_secs=5, interval_secs=1) @@ -43,7 +48,9 @@ job = Service( role = getpass.getuser(), environment = 'test', contact = '{{role}}@localhost', - announce = Announcer(), + announce = Announcer( + portmap={'alias': 'http'}, + ), ) jobs = [ http://git-wip-us.apache.org/repos/asf/aurora/blob/915459da/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh ---------------------------------------------------------------------- diff --git a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh index 3471756..eee6b4c 100755 --- a/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh +++ b/src/test/sh/org/apache/aurora/e2e/test_end_to_end.sh @@ -266,12 +266,55 @@ test_quota() { aurora quota get $_cluster/$_role } +test_discovery_info() { + local _task_id_prefix=$1 + local _discovery_name=$2 + + if ! [[ -x "$(command -v jq)" ]]; then + echo "jq is not installed, skipping discovery info test" + return 0 + fi + + framework_info=$(curl --silent '192.168.33.7:5050/state' | jq '.frameworks | map(select(.name == "TwitterScheduler"))') + if [[ -z $framework_info ]]; then + echo "Cannot get framework info for $framework" + exit 1 + fi + + task_info=$(echo $framework_info | jq --arg task_id_prefix "${_task_id_prefix}" '.[0]["tasks"] | map(select(.id | contains($task_id_prefix)))') + if [[ -z $task_info ]]; then + echo "Cannot get task blob json for task id prefix ${_task_id_prefix}" + exit 1 + fi + + discovery_info=$(echo $task_info | jq '.[0]["discovery"]') + if [[ -z $discovery_info ]]; then + echo "Cannot get discovery info json from task blob ${task_blob}" + exit 1 + fi + + name=$(echo $discovery_info | jq '.["name"]') + if [[ "$name" -ne "\"$_discovery_name\"" ]]; then + echo "discovery info name $name does not equal to expected \"$_discovery_name\"" + exit 1 + fi + + num_ports=$(echo $discovery_info | jq '.["ports"]["ports"] | length') + + if ! [[ "$num_ports" -gt 0 ]]; then + echo "num of ports in discovery info is $num_ports which is not greater than zero" + exit 1 + fi +} + test_http_example() { local _cluster=$1 _role=$2 _env=$3 local _base_config=$4 _updated_config=$5 local _bad_healthcheck_config=$6 local _job=$7 local _jobkey="$_cluster/$_role/$_env/$_job" + local _task_id_prefix="${_role}-${_env}-${_job}-0" + local _discovery_name="${_job}.${_env}.${_role}" test_config $_base_config $_jobkey test_inspect $_jobkey $_base_config @@ -279,6 +322,7 @@ test_http_example() { test_job_status $_cluster $_role $_env $_job test_scheduler_ui $_role $_env $_job test_observer_ui $_cluster $_role $_job + test_discovery_info $_task_id_prefix $_discovery_name test_restart $_jobkey test_update $_jobkey $_updated_config $_cluster test_update_fail $_jobkey $_base_config $_cluster $_bad_healthcheck_config
