METRON-859 Use REST application with Kerberos (merrimanr) closes apache/incubator-metron#535
Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/5dd87886 Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/5dd87886 Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/5dd87886 Branch: refs/heads/Metron_0.4.0 Commit: 5dd87886d6288b3f65e4f26e88ef10dcc82895fe Parents: ffe2d93 Author: merrimanr <[email protected]> Authored: Wed Apr 26 14:01:02 2017 -0500 Committer: merrimanr <[email protected]> Committed: Wed Apr 26 14:01:02 2017 -0500 ---------------------------------------------------------------------- dependencies_with_url.csv | 2 + metron-interface/metron-rest/README.md | 32 ++++++- metron-interface/metron-rest/pom.xml | 6 ++ .../apache/metron/rest/MetronRestConstants.java | 4 + .../apache/metron/rest/config/HadoopConfig.java | 18 +++- .../apache/metron/rest/config/KafkaConfig.java | 11 ++- .../metron/rest/config/RestTemplateConfig.java | 20 ++++- .../rest/service/impl/StormCLIWrapper.java | 15 ++++ .../src/main/resources/application-vagrant.yml | 6 ++ .../metron/rest/config/HadoopConfigTest.java | 89 ++++++++++++++++++++ .../rest/config/RestTemplateConfigTest.java | 67 +++++++++++++++ .../rest/service/impl/StormCLIWrapperTest.java | 24 ++++++ .../metron-rest/src/test/resources/README.vm | 32 ++++++- 13 files changed, 313 insertions(+), 13 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/dependencies_with_url.csv ---------------------------------------------------------------------- diff --git a/dependencies_with_url.csv b/dependencies_with_url.csv index 25650a3..053f11a 100644 --- a/dependencies_with_url.csv +++ b/dependencies_with_url.csv @@ -293,3 +293,5 @@ org.atteo.classindex:classindex:jar:3.3:compile,ASLv2,https://github.com/atteo/c com.squareup.okhttp:okhttp:jar:2.4.0:compile,ASLv2,https://github.com/square/okhttp com.squareup.okio:okio:jar:1.4.0:compile,ASLv2,https://github.com/square/okhttp org.htrace:htrace-core:jar:3.0.4:compile,ASLv2,http://htrace.incubator.apache.org/ +org.springframework.security.kerberos:spring-security-kerberos-client:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos +org.springframework.security.kerberos:spring-security-kerberos-core:jar:1.0.1.RELEASE:compile,ASLv2,https://github.com/spring-projects/spring-security-kerberos http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/README.md ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/README.md b/metron-interface/metron-rest/README.md index c52b21d..89984ed 100644 --- a/metron-interface/metron-rest/README.md +++ b/metron-interface/metron-rest/README.md @@ -22,7 +22,7 @@ This module provides a RESTful API for interacting with Metron. metron-rest-$METRON_VERSION.jar ``` -1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). +1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). 1. Start the application with this command: ``` @@ -518,11 +518,35 @@ The metron-rest application will be available at http://localhost:8080/swagger-u To run the application locally on the Quick Dev host, package the application and scp the archive to node1: ``` mvn clean package -scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:~/ +scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME ``` -Login to node1 and unarchive the metron-rest application. Start the application on a different port to avoid conflicting with Ambari: +Login to node1 and unarchive the metron-rest application: ``` -java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 +ssh root@node1 +cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz +``` +Start the application on a different port to avoid conflicting with Ambari: +``` +java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 +``` +In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls: +``` +sudo su - +export ZOOKEEPER=node1 +export BROKERLIST=node1 +export HDP_HOME="/usr/hdp/current" +export METRON_VERSION="0.4.0" +export METRON_HOME="/usr/metron/${METRON_VERSION}" +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest +``` + +Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support: +``` +su metron +cd ~ +java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true ``` The metron-rest application will be available at http://node1:8082/swagger-ui.html#/. http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/pom.xml ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/pom.xml b/metron-interface/metron-rest/pom.xml index 421b7fe..cc53192 100644 --- a/metron-interface/metron-rest/pom.xml +++ b/metron-interface/metron-rest/pom.xml @@ -29,6 +29,7 @@ <curator.version>2.7.1</curator.version> <powermock.version>1.6.4</powermock.version> <spring.boot.version>1.4.1.RELEASE</spring.boot.version> + <spring.kerberos.version>1.0.1.RELEASE</spring.kerberos.version> <swagger.version>2.5.0</swagger.version> <mysql.client.version>5.1.40</mysql.client.version> <emma.version>1.0-alpha-3</emma.version> @@ -57,6 +58,11 @@ <artifactId>spring-boot-starter-security</artifactId> </dependency> <dependency> + <groupId>org.springframework.security.kerberos</groupId> + <artifactId>spring-security-kerberos-client</artifactId> + <version>${spring.kerberos.version}</version> + </dependency> + <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-jdbc</artifactId> </dependency> http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java index 8786222..2a147e6 100644 --- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java +++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/MetronRestConstants.java @@ -52,4 +52,8 @@ public class MetronRestConstants { public static final String HDFS_URL_SPRING_PROPERTY = "hdfs.namenode.url"; public static final String DEFAULT_HDFS_URL = "file:///"; + + public static final String KERBEROS_ENABLED_SPRING_PROPERTY = "kerberos.enabled"; + public static final String KERBEROS_PRINCIPLE_SPRING_PROPERTY = "kerberos.principal"; + public static final String KERBEROS_KEYTAB_SPRING_PROPERTY = "kerberos.keytab"; } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java index f4b8bdd..7b1bf2f 100644 --- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java +++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/HadoopConfig.java @@ -17,22 +17,36 @@ */ package org.apache.metron.rest.config; +import org.apache.hadoop.security.UserGroupInformation; import org.apache.metron.rest.MetronRestConstants; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.core.env.Environment; +import java.io.IOException; + @Configuration public class HadoopConfig { - @Autowired private Environment environment; + @Autowired + public HadoopConfig(Environment environment) { + this.environment = environment; + } + @Bean - public org.apache.hadoop.conf.Configuration configuration() { + public org.apache.hadoop.conf.Configuration configuration() throws IOException { org.apache.hadoop.conf.Configuration configuration = new org.apache.hadoop.conf.Configuration(); configuration.set("fs.defaultFS", environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL)); + if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) { + configuration.set("hadoop.security.authentication", "KERBEROS"); + UserGroupInformation.setConfiguration(configuration); + String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY); + String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY); + UserGroupInformation.loginUserFromKeytab(userPrincipal, keyTabLocation); + } return configuration; } } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java index f6ff73c..309a549 100644 --- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java +++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/KafkaConfig.java @@ -36,10 +36,14 @@ import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE; @Profile("!" + TEST_PROFILE) public class KafkaConfig { - @Autowired private Environment environment; @Autowired + public KafkaConfig(Environment environment) { + this.environment = environment; + } + + @Autowired private ZkClient zkClient; @Bean @@ -51,12 +55,15 @@ public class KafkaConfig { public KafkaConsumer<String, String> kafkaConsumer() { Properties props = new Properties(); props.put("bootstrap.servers", environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY)); - props.put("group.id", "metron-config"); + props.put("group.id", "metron-rest"); props.put("enable.auto.commit", "false"); props.put("auto.commit.interval.ms", "1000"); props.put("session.timeout.ms", "30000"); props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); + if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) { + props.put("security.protocol", "SASL_PLAINTEXT"); + } return new KafkaConsumer<>(props); } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java index 8fea90c..7443ccd 100644 --- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java +++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/config/RestTemplateConfig.java @@ -17,9 +17,13 @@ */ package org.apache.metron.rest.config; +import org.apache.metron.rest.MetronRestConstants; +import org.springframework.beans.factory.annotation.Autowired; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.springframework.context.annotation.Profile; +import org.springframework.core.env.Environment; +import org.springframework.security.kerberos.client.KerberosRestTemplate; import org.springframework.web.client.RestTemplate; import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE; @@ -28,9 +32,23 @@ import static org.apache.metron.rest.MetronRestConstants.TEST_PROFILE; @Profile("!" + TEST_PROFILE) public class RestTemplateConfig { + private Environment environment; + + @Autowired + public RestTemplateConfig(Environment environment) { + this.environment = environment; + } + @Bean public RestTemplate restTemplate() { - return new RestTemplate(); + if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) { + String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY); + String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY); + return new KerberosRestTemplate(keyTabLocation, userPrincipal); + } else { + return new RestTemplate(); + } + } } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java index c5569c5..1158721 100644 --- a/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java +++ b/metron-interface/metron-rest/src/main/java/org/apache/metron/rest/service/impl/StormCLIWrapper.java @@ -43,26 +43,32 @@ public class StormCLIWrapper { } public int startParserTopology(String name) throws RestException { + kinit(); return runCommand(getParserStartCommand(name)); } public int stopParserTopology(String name, boolean stopNow) throws RestException { + kinit(); return runCommand(getStopCommand(name, stopNow)); } public int startEnrichmentTopology() throws RestException { + kinit(); return runCommand(getEnrichmentStartCommand()); } public int stopEnrichmentTopology(boolean stopNow) throws RestException { + kinit(); return runCommand(getStopCommand(ENRICHMENT_TOPOLOGY_NAME, stopNow)); } public int startIndexingTopology() throws RestException { + kinit(); return runCommand(getIndexingStartCommand()); } public int stopIndexingTopology(boolean stopNow) throws RestException { + kinit(); return runCommand(getStopCommand(INDEXING_TOPOLOGY_NAME, stopNow)); } @@ -150,5 +156,14 @@ public class StormCLIWrapper { return stormClientVersionInstalled; } + protected void kinit() throws RestException { + if (environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)) { + String keyTabLocation = environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY); + String userPrincipal = environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY); + String[] kinitCommand = {"kinit", "-kt", keyTabLocation, userPrincipal}; + runCommand(kinitCommand); + } + } + } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/main/resources/application-vagrant.yml ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml index 76bef56..158886f 100644 --- a/metron-interface/metron-rest/src/main/resources/application-vagrant.yml +++ b/metron-interface/metron-rest/src/main/resources/application-vagrant.yml @@ -49,3 +49,9 @@ storm: script.path: /usr/metron/${metron.version}/bin/start_enrichment_topology.sh indexing: script.path: /usr/metron/${metron.version}/bin/start_elasticsearch_topology.sh + +kerberos: + enabled: false + principal: [email protected] + keytab: /etc/security/keytabs/metron.headless.keytab + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java new file mode 100644 index 0000000..c262783 --- /dev/null +++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/HadoopConfigTest.java @@ -0,0 +1,89 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.rest.config; + +import org.apache.hadoop.conf.Configuration; +import org.apache.hadoop.security.UserGroupInformation; +import org.apache.metron.rest.MetronRestConstants; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.powermock.core.classloader.annotations.PrepareForTest; +import org.powermock.modules.junit4.PowerMockRunner; +import org.springframework.core.env.Environment; + +import java.io.IOException; + +import static org.junit.Assert.assertEquals; +import static org.mockito.Matchers.any; +import static org.mockito.Matchers.anyString; +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.never; +import static org.mockito.Mockito.reset; +import static org.mockito.Mockito.when; + +import static org.powermock.api.mockito.PowerMockito.mockStatic; +import static org.powermock.api.mockito.PowerMockito.verifyStatic; + +@RunWith(PowerMockRunner.class) +@PrepareForTest({HadoopConfig.class, UserGroupInformation.class}) +public class HadoopConfigTest { + + private Environment environment; + private HadoopConfig hadoopConfig; + + @Before + public void setUp() throws Exception { + environment = mock(Environment.class); + hadoopConfig = new HadoopConfig(environment); + mockStatic(UserGroupInformation.class); + } + + @Test + public void configurationShouldReturnProperKerberosConfiguration() throws IOException { + when(environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL)).thenReturn("default filesystem"); + when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation"); + when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal"); + + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true); + + Configuration configuration = hadoopConfig.configuration(); + + verifyStatic(); + UserGroupInformation.setConfiguration(any(Configuration.class)); + UserGroupInformation.loginUserFromKeytab("metron keytabLocation", "metron principal"); + + assertEquals("default filesystem", configuration.get("fs.defaultFS")); + assertEquals("KERBEROS", configuration.get("hadoop.security.authentication")); + } + + @Test + public void configurationShouldReturnProperConfiguration() throws IOException { + when(environment.getProperty(MetronRestConstants.HDFS_URL_SPRING_PROPERTY, MetronRestConstants.DEFAULT_HDFS_URL)).thenReturn("default filesystem"); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); + + Configuration configuration = hadoopConfig.configuration(); + + verifyStatic(never()); + UserGroupInformation.setConfiguration(any(Configuration.class)); + UserGroupInformation.loginUserFromKeytab(anyString(), anyString()); + + assertEquals("default filesystem", configuration.get("fs.defaultFS")); + assertEquals("simple", configuration.get("hadoop.security.authentication")); + } +} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java new file mode 100644 index 0000000..0d9d620 --- /dev/null +++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/config/RestTemplateConfigTest.java @@ -0,0 +1,67 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.metron.rest.config; + +import org.apache.metron.rest.MetronRestConstants; +import org.junit.Before; +import org.junit.Test; +import org.junit.runner.RunWith; +import org.powermock.core.classloader.annotations.PrepareForTest; +import org.powermock.modules.junit4.PowerMockRunner; +import org.springframework.core.env.Environment; +import org.springframework.security.kerberos.client.KerberosRestTemplate; +import org.springframework.web.client.RestTemplate; + +import static org.mockito.Mockito.mock; +import static org.mockito.Mockito.when; +import static org.powermock.api.mockito.PowerMockito.verifyNew; +import static org.powermock.api.mockito.PowerMockito.whenNew; + +@RunWith(PowerMockRunner.class) +@PrepareForTest({RestTemplateConfig.class, KerberosRestTemplate.class, RestTemplate.class}) +public class RestTemplateConfigTest { + + private Environment environment; + private RestTemplateConfig restTemplateConfig; + + @Before + public void setUp() throws Exception { + environment = mock(Environment.class); + restTemplateConfig = new RestTemplateConfig(environment); + } + + @Test + public void restTemplateShouldReturnProperTemplate() throws Exception { + when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation"); + when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal"); + + whenNew(KerberosRestTemplate.class).withParameterTypes(String.class, String.class).withArguments("metron keytabLocation", "metron principal").thenReturn(null); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true); + + restTemplateConfig.restTemplate(); + verifyNew(KerberosRestTemplate.class).withArguments("metron keytabLocation", "metron principal"); + + whenNew(RestTemplate.class).withNoArguments().thenReturn(null); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); + + restTemplateConfig.restTemplate(); + verifyNew(RestTemplate.class).withNoArguments(); + } + + +} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java index cb26783..9fb067a 100644 --- a/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java +++ b/metron-interface/metron-rest/src/test/java/org/apache/metron/rest/service/impl/StormCLIWrapperTest.java @@ -24,6 +24,7 @@ import org.junit.Rule; import org.junit.Test; import org.junit.rules.ExpectedException; import org.junit.runner.RunWith; +import org.mockito.Mockito; import org.powermock.core.classloader.annotations.PrepareForTest; import org.powermock.modules.junit4.PowerMockRunner; import org.springframework.core.env.Environment; @@ -37,6 +38,7 @@ import java.util.Map; import static java.nio.charset.StandardCharsets.UTF_8; import static org.junit.Assert.assertEquals; import static org.mockito.Matchers.anyVararg; +import static org.mockito.Mockito.times; import static org.mockito.Mockito.verify; import static org.powermock.api.mockito.PowerMockito.mock; import static org.powermock.api.mockito.PowerMockito.verifyNew; @@ -73,6 +75,7 @@ public class StormCLIWrapperTest { when(environment.getProperty(MetronRestConstants.PARSER_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_parser"); when(environment.getProperty(MetronRestConstants.KAFKA_BROKER_URL_SPRING_PROPERTY)).thenReturn("kafka_broker_url"); when(environment.getProperty(MetronRestConstants.ZK_URL_SPRING_PROPERTY)).thenReturn("zookeeper_url"); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.startParserTopology("bro")); @@ -85,6 +88,7 @@ public class StormCLIWrapperTest { whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder); when(processBuilder.start()).thenReturn(process); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.stopParserTopology("bro", false)); @@ -97,6 +101,7 @@ public class StormCLIWrapperTest { whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder); when(processBuilder.start()).thenReturn(process); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.stopParserTopology("bro", true)); @@ -110,6 +115,7 @@ public class StormCLIWrapperTest { when(processBuilder.start()).thenReturn(process); when(environment.getProperty(MetronRestConstants.ENRICHMENT_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_enrichment"); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.startEnrichmentTopology()); @@ -123,6 +129,7 @@ public class StormCLIWrapperTest { whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder); when(processBuilder.start()).thenReturn(process); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.stopEnrichmentTopology(false)); @@ -136,6 +143,7 @@ public class StormCLIWrapperTest { when(processBuilder.start()).thenReturn(process); when(environment.getProperty(MetronRestConstants.INDEXING_SCRIPT_PATH_SPRING_PROPERTY)).thenReturn("/start_indexing"); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.startIndexingTopology()); @@ -149,6 +157,7 @@ public class StormCLIWrapperTest { whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder); when(processBuilder.start()).thenReturn(process); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(false); when(process.exitValue()).thenReturn(0); assertEquals(0, stormCLIWrapper.stopIndexingTopology(false)); @@ -213,4 +222,19 @@ public class StormCLIWrapperTest { stormCLIWrapper.stormClientVersionInstalled(); } + + @Test + public void kinitShouldRunCommandProperly() throws Exception { + whenNew(ProcessBuilder.class).withParameterTypes(String[].class).withArguments(anyVararg()).thenReturn(processBuilder); + + when(processBuilder.start()).thenReturn(process); + when(environment.getProperty(MetronRestConstants.KERBEROS_ENABLED_SPRING_PROPERTY, Boolean.class, false)).thenReturn(true); + when(environment.getProperty(MetronRestConstants.KERBEROS_KEYTAB_SPRING_PROPERTY)).thenReturn("metron keytabLocation"); + when(environment.getProperty(MetronRestConstants.KERBEROS_PRINCIPLE_SPRING_PROPERTY)).thenReturn("metron principal"); + when(process.exitValue()).thenReturn(0); + + stormCLIWrapper.kinit(); + verify(process, times(1)).waitFor(); + verifyNew(ProcessBuilder.class).withArguments("kinit", "-kt", "metron keytabLocation", "metron principal"); + } } http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/5dd87886/metron-interface/metron-rest/src/test/resources/README.vm ---------------------------------------------------------------------- diff --git a/metron-interface/metron-rest/src/test/resources/README.vm b/metron-interface/metron-rest/src/test/resources/README.vm index ca54e6a..cff7ca7 100644 --- a/metron-interface/metron-rest/src/test/resources/README.vm +++ b/metron-interface/metron-rest/src/test/resources/README.vm @@ -22,7 +22,7 @@ This module provides a RESTful API for interacting with Metron. metron-rest-$METRON_VERSION.jar ``` -1. Create an `application.yml` file with the contents of [application-docker.yml](src/main/resources/application-docker.yml). Substitute the appropriate Metron service urls (Kafka, Zookeeper, Storm, etc.) in properties containing `${docker.host.address}` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). +1. Create an `application.yml` file with the contents of [application-vagrant.yml](src/main/resources/application-vagrant.yml). Substitute the appropriate Metron service hosts (Kafka, Zookeeper, Storm, etc.) in properties containing `node1` and update the `spring.datasource.*` properties as needed (see the [Security](#security) section for more details). 1. Start the application with this command: ``` @@ -128,11 +128,35 @@ The metron-rest application will be available at http://localhost:8080/swagger-u To run the application locally on the Quick Dev host, package the application and scp the archive to node1: ``` mvn clean package -scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:~/ +scp ./target/metron-rest-$METRON_VERSION-archive.tar.gz root@node1:$METRON_HOME ``` -Login to node1 and unarchive the metron-rest application. Start the application on a different port to avoid conflicting with Ambari: +Login to node1 and unarchive the metron-rest application: ``` -java -jar ./lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 +ssh root@node1 +cd $METRON_HOME && tar xf ./metron-rest-$METRON_VERSION-archive.tar.gz +``` +Start the application on a different port to avoid conflicting with Ambari: +``` +java -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 +``` +In a cluster with Kerberos enabled, first add metron-rest to the Kafka acls: +``` +sudo su - +export ZOOKEEPER=node1 +export BROKERLIST=node1 +export HDP_HOME="/usr/hdp/current" +export METRON_VERSION="0.4.0" +export METRON_HOME="/usr/metron/${METRON_VERSION}" +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic ambari_kafka_service_check +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --topic __consumer_offsets +${HDP_HOME}/kafka-broker/bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=${ZOOKEEPER}:2181 --add --allow-principal User:metron --group metron-rest +``` + +Then start the application as the metron user while including references to the jaas and krb5.confg files and enabling kerberos support: +``` +su metron +cd ~ +java -Djava.security.auth.login.config=/home/metron/.storm/client_jaas.conf -Djava.security.krb5.conf=/etc/krb5.conf -jar $METRON_HOME/lib/metron-rest-$METRON_VERSION.jar --spring.profiles.active=vagrant,dev --server.port=8082 --kerberos.enabled=true ``` The metron-rest application will be available at http://node1:8082/swagger-ui.html#/.
