METRON-1652 Document X-Pack Common Problem (nickwallen) closes apache/metron#1092
Project: http://git-wip-us.apache.org/repos/asf/metron/repo Commit: http://git-wip-us.apache.org/repos/asf/metron/commit/4eb9a8eb Tree: http://git-wip-us.apache.org/repos/asf/metron/tree/4eb9a8eb Diff: http://git-wip-us.apache.org/repos/asf/metron/diff/4eb9a8eb Branch: refs/heads/feature/METRON-1554-pcap-query-panel Commit: 4eb9a8ebca733bbae96e7e19e9064c5e71666884 Parents: d90f7cf Author: nickwallen <n...@nickallen.org> Authored: Thu Jul 5 11:07:59 2018 -0400 Committer: nickallen <nickal...@apache.org> Committed: Thu Jul 5 11:07:59 2018 -0400 ---------------------------------------------------------------------- metron-deployment/Kerberos-manual-setup.md | 441 +++++++++++++----------- 1 file changed, 243 insertions(+), 198 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/metron/blob/4eb9a8eb/metron-deployment/Kerberos-manual-setup.md ---------------------------------------------------------------------- diff --git a/metron-deployment/Kerberos-manual-setup.md b/metron-deployment/Kerberos-manual-setup.md index ecda858..e3de148 100644 --- a/metron-deployment/Kerberos-manual-setup.md +++ b/metron-deployment/Kerberos-manual-setup.md @@ -538,203 +538,248 @@ In order to correct this, you should: X-Pack ------ -First, stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. +1. Stop the random_access_indexing topology through the Storm UI or from the CLI, e.g. -``` -storm kill random_access_indexing -``` - -Here are instructions for enabling X-Pack with Elasticsearch and Kibana: https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html - -You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials. - -``` -sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser -``` - -Once you've picked a password to connect to ES, you need to upload a 1-line file to HDFS with that password in it. Metron will use this file to securely read the password in order to connect to ES securely. - -Here is an example using "changeme" as the password - -``` -echo changeme > /tmp/xpack-password -sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/ -sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/ -sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password -``` - -New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below. - -Add the es settings to global.json - -``` -/usr/metron/0.5.1/config/zookeeper/global.json -> - - "es.client.settings" : { - "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient", - "es.xpack.username" : "transport_client_user", - "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password" - } -``` - -Submit the update to Zookeeper - -``` -$METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER -``` - -The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help. - -``` -<?xml version="1.0" encoding="UTF-8"?> -<!-- - Licensed to the Apache Software - Foundation (ASF) under one or more contributor license agreements. See the - NOTICE file distributed with this work for additional information regarding - copyright ownership. The ASF licenses this file to You under the Apache License, - Version 2.0 (the "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 - Unless required by applicable law or agreed to in writing, software distributed - under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES - OR CONDITIONS OF ANY KIND, either express or implied. See the License for - the specific language governing permissions and limitations under the License. - --> -<project xmlns="http://maven.apache.org/POM/4.0.0"; - xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; - xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> - <modelVersion>4.0.0</modelVersion> - <groupId>org.elasticsearch</groupId> - <artifactId>elasticsearch-xpack-shaded</artifactId> - <name>elasticsearch-xpack-shaded</name> - <packaging>jar</packaging> - <version>5.6.2</version> - <repositories> - <repository> - <id>elasticsearch-releases</id> - <url>https://artifacts.elastic.co/maven</url> - <releases> - <enabled>true</enabled> - </releases> - <snapshots> - <enabled>false</enabled> - </snapshots> - </repository> - </repositories> - <dependencies> - <dependency> - <groupId>org.elasticsearch.client</groupId> - <artifactId>x-pack-transport</artifactId> - <version>5.6.2</version> - <exclusions> - <exclusion> - <groupId>com.fasterxml.jackson.dataformat</groupId> - <artifactId>jackson-dataformat-yaml</artifactId> - </exclusion> - <exclusion> - <groupId>com.fasterxml.jackson.dataformat</groupId> - <artifactId>jackson-dataformat-cbor</artifactId> - </exclusion> - <exclusion> - <groupId>com.fasterxml.jackson.core</groupId> - <artifactId>jackson-core</artifactId> - </exclusion> - <exclusion> - <groupId>org.slf4j</groupId> - <artifactId>slf4j-api</artifactId> - </exclusion> - <exclusion> - <groupId>org.slf4j</groupId> - <artifactId>slf4j-log4j12</artifactId> - </exclusion> - <exclusion> - <groupId>log4j</groupId> - <artifactId>log4j</artifactId> - </exclusion> - <exclusion> <!-- this is causing a weird build error if not excluded - Error creating shaded jar: null: IllegalArgumentException --> - <groupId>org.apache.logging.log4j</groupId> - <artifactId>log4j-api</artifactId> - </exclusion> - </exclusions> - </dependency> - </dependencies> - <build> - <plugins> - <plugin> - <groupId>org.apache.maven.plugins</groupId> - <artifactId>maven-shade-plugin</artifactId> - <version>2.4.3</version> - <configuration> - <createDependencyReducedPom>true</createDependencyReducedPom> - </configuration> - <executions> - <execution> - <phase>package</phase> - <goals> - <goal>shade</goal> - </goals> - <configuration> - <filters> - <filter> - <artifact>*:*</artifact> - <excludes> - <exclude>META-INF/*.SF</exclude> - <exclude>META-INF/*.DSA</exclude> - <exclude>META-INF/*.RSA</exclude> - </excludes> - </filter> - </filters> - <relocations> - <relocation> - <pattern>io.netty</pattern> - <shadedPattern>org.apache.metron.io.netty</shadedPattern> - </relocation> - <relocation> - <pattern>org.apache.logging.log4j</pattern> - <shadedPattern>org.apache.metron.logging.log4j</shadedPattern> - </relocation> - </relocations> - <artifactSet> - <excludes> - <exclude>org.slf4j.impl*</exclude> - <exclude>org.slf4j:slf4j-log4j*</exclude> - </excludes> - </artifactSet> - <transformers> - <transformer - implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer"> - <resources> - <resource>.yaml</resource> - <resource>LICENSE.txt</resource> - <resource>ASL2.0</resource> - <resource>NOTICE.txt</resource> - </resources> - </transformer> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> - <transformer - implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> - <mainClass></mainClass> - </transformer> - </transformers> - </configuration> - </execution> - </executions> - </plugin> - </plugins> - </build> -</project> -``` - -Once you've built the elasticsearch-xpack-shaded-5.6.2.jar, it needs to be made available to Storm when you submit the topology. Create a contrib directory for indexing and put the jar file in this directory. - -``` -/usr/metron/0.5.1/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar -``` - -Now you can restart the Elasticsearch topology. Note, you should perform this step manually, as follows. - -``` -$METRON_HOME/bin/start_elasticsearch_topology.sh -``` + ``` + storm kill random_access_indexing + ``` -Once you've performed these steps, you shoud be able to start seeing data in your ES indexes. +1. [Follow these instructions](https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html) to enable the X-Pack with Elasticsearch and Kibana. The X-Pack plugin will need installed on each of the nodes running Elasticsearch. + +1. You need to be sure to add the appropriate username and password for Elasticsearch and Kibana to enable external connections from Metron components. e.g. the following will create a user "transport_client_user" with password "changeme" and "superuser" credentials. + + ``` + sudo /usr/share/elasticsearch/bin/x-pack/users useradd transport_client_user -p changeme -r superuser + ``` + +1. Once you've picked a password to connect to ES, you need to upload a 1-line file to HDFS with that password in it. Metron will use this file to securely read the password in order to connect to ES securely. + + Here is an example using "changeme" as the password + + ``` + echo changeme > /tmp/xpack-password + sudo -u hdfs hdfs dfs -mkdir /apps/metron/elasticsearch/ + sudo -u hdfs hdfs dfs -put /tmp/xpack-password /apps/metron/elasticsearch/ + sudo -u hdfs hdfs dfs -chown metron:metron /apps/metron/elasticsearch/xpack-password + ``` + +1. New settings have been added to configure the Elasticsearch client. By default the client will run as the normal ES prebuilt transport client. If you enable X-Pack you should set the es.client.class as shown below. + + Add the `es.client.settings` to global.json + + ``` + /usr/metron/0.5.1/config/zookeeper/global.json -> + + "es.client.settings" : { + "es.client.class" : "org.elasticsearch.xpack.client.PreBuiltXPackTransportClient", + "es.xpack.username" : "transport_client_user", + "es.xpack.password.file" : "/apps/metron/elasticsearch/xpack-password" + } + ``` + + Submit the update to Zookeeper + + ``` + $METRON_HOME/bin/zk_load_configs.sh -m PUSH -i METRON_HOME/config/zookeeper/ -z $ZOOKEEPER + ``` + +1. The last step before restarting the topology is to create a custom X-Pack shaded and relocated jar. This is up to you because of licensing restrictions, but here is a sample Maven pom file that should help. + + ``` + <?xml version="1.0" encoding="UTF-8"?> + <!-- + Licensed to the Apache Software + Foundation (ASF) under one or more contributor license agreements. See the + NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to You under the Apache License, + Version 2.0 (the "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 + Unless required by applicable law or agreed to in writing, software distributed + under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES + OR CONDITIONS OF ANY KIND, either express or implied. See the License for + the specific language governing permissions and limitations under the License. + --> + <project xmlns="http://maven.apache.org/POM/4.0.0"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd";> + <modelVersion>4.0.0</modelVersion> + <groupId>org.elasticsearch</groupId> + <artifactId>elasticsearch-xpack-shaded</artifactId> + <name>elasticsearch-xpack-shaded</name> + <packaging>jar</packaging> + <version>5.6.2</version> + <repositories> + <repository> + <id>elasticsearch-releases</id> + <url>https://artifacts.elastic.co/maven</url> + <releases> + <enabled>true</enabled> + </releases> + <snapshots> + <enabled>false</enabled> + </snapshots> + </repository> + </repositories> + <dependencies> + <dependency> + <groupId>org.elasticsearch.client</groupId> + <artifactId>x-pack-transport</artifactId> + <version>5.6.2</version> + <exclusions> + <exclusion> + <groupId>com.fasterxml.jackson.dataformat</groupId> + <artifactId>jackson-dataformat-yaml</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.dataformat</groupId> + <artifactId>jackson-dataformat-cbor</artifactId> + </exclusion> + <exclusion> + <groupId>com.fasterxml.jackson.core</groupId> + <artifactId>jackson-core</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-api</artifactId> + </exclusion> + <exclusion> + <groupId>org.slf4j</groupId> + <artifactId>slf4j-log4j12</artifactId> + </exclusion> + <exclusion> + <groupId>log4j</groupId> + <artifactId>log4j</artifactId> + </exclusion> + <exclusion> <!-- this is causing a weird build error if not excluded - Error creating shaded jar: null: IllegalArgumentException --> + <groupId>org.apache.logging.log4j</groupId> + <artifactId>log4j-api</artifactId> + </exclusion> + </exclusions> + </dependency> + </dependencies> + <build> + <plugins> + <plugin> + <groupId>org.apache.maven.plugins</groupId> + <artifactId>maven-shade-plugin</artifactId> + <version>2.4.3</version> + <configuration> + <createDependencyReducedPom>true</createDependencyReducedPom> + </configuration> + <executions> + <execution> + <phase>package</phase> + <goals> + <goal>shade</goal> + </goals> + <configuration> + <filters> + <filter> + <artifact>*:*</artifact> + <excludes> + <exclude>META-INF/*.SF</exclude> + <exclude>META-INF/*.DSA</exclude> + <exclude>META-INF/*.RSA</exclude> + </excludes> + </filter> + </filters> + <relocations> + <relocation> + <pattern>io.netty</pattern> + <shadedPattern>org.apache.metron.io.netty</shadedPattern> + </relocation> + <relocation> + <pattern>org.apache.logging.log4j</pattern> + <shadedPattern>org.apache.metron.logging.log4j</shadedPattern> + </relocation> + </relocations> + <artifactSet> + <excludes> + <exclude>org.slf4j.impl*</exclude> + <exclude>org.slf4j:slf4j-log4j*</exclude> + </excludes> + </artifactSet> + <transformers> + <transformer + implementation="org.apache.maven.plugins.shade.resource.DontIncludeResourceTransformer"> + <resources> + <resource>.yaml</resource> + <resource>LICENSE.txt</resource> + <resource>ASL2.0</resource> + <resource>NOTICE.txt</resource> + </resources> + </transformer> + <transformer + implementation="org.apache.maven.plugins.shade.resource.ServicesResourceTransformer"/> + <transformer + implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> + <mainClass></mainClass> + </transformer> + </transformers> + </configuration> + </execution> + </executions> + </plugin> + </plugins> + </build> + </project> + ``` + +1. Once you've built the `elasticsearch-xpack-shaded-5.6.2.jar`, it needs to be made available to Storm when you submit the topology. Create a contrib directory for indexing and put the jar file in this directory. + + ``` + /usr/metron/0.5.1/indexing_contrib/elasticsearch-xpack-shaded-5.6.2.jar + ``` + +1. Now you can restart the Elasticsearch topology. Note, you should perform this step manually, as follows. + + ``` + $METRON_HOME/bin/start_elasticsearch_topology.sh + ``` + +Once you've performed these steps, you should be able to start seeing data in your ES indexes. + +### X-Pack Common Problems + +#### java.io.FileNotFoundException: File /apps/metron/elasticsearch/xpack-password does not exist + +#### Problem + +The random access indexer topology fails with the following exception. This exception might occur on only some of the Storm worker nodes. + + ``` + 2018-07-02 10:24:08.267 o.a.s.util Thread-8-indexingBolt-executor[3 3] [ERROR] Async loop died! + java.lang.RuntimeException: java.lang.IllegalArgumentException: Unable to read XPack password file from HDFS location '/apps/metron/elasticsearch/xpack-password' + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:201) ~[stormjar.jar:?] + at org.apache.storm.daemon.executor$fn__10195$fn__10208.invoke(executor.clj:800) ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] + at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:482) [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] + at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] + at java.lang.Thread.run(Thread.java:748) [?:1.8.0_162] + Caused by: java.lang.IllegalArgumentException: Unable to read XPack password file from HDFS location '/apps/metron/elasticsearch/xpack-password' + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getPasswordFromFile(ElasticsearchUtils.java:201) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.setXPackSecurityOrNone(ElasticsearchUtils.java:187) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getClient(ElasticsearchUtils.java:147) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.init(ElasticsearchWriter.java:53) ~[stormjar.jar:?] + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:199) ~[stormjar.jar:?] + ... 4 more + Caused by: java.io.FileNotFoundException: File /apps/metron/elasticsearch/xpack-password does not exist + at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:606) ~[stormjar.jar:?] + at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:819) ~[stormjar.jar:?] + at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:596) ~[stormjar.jar:?] + at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:421) ~[stormjar.jar:?] + at org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:140) ~[stormjar.jar:?] + at org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:341) ~[stormjar.jar:?] + at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:767) ~[stormjar.jar:?] + at org.apache.metron.common.utils.HDFSUtils.readFile(HDFSUtils.java:55) ~[stormjar.jar:?] + at org.apache.metron.common.utils.HDFSUtils.readFile(HDFSUtils.java:40) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getPasswordFromFile(ElasticsearchUtils.java:198) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.setXPackSecurityOrNone(ElasticsearchUtils.java:187) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.utils.ElasticsearchUtils.getClient(ElasticsearchUtils.java:147) ~[stormjar.jar:?] + at org.apache.metron.elasticsearch.writer.ElasticsearchWriter.init(ElasticsearchWriter.java:53) ~[stormjar.jar:?] + at org.apache.metron.writer.bolt.BulkMessageWriterBolt.prepare(BulkMessageWriterBolt.java:199) ~[stormjar.jar:?] + ... 4 more + ``` + +#### Solution + +This can occur when an HDFS Client is not installed on the Storm worker nodes. This might occur on any Storm worker node where an HDFS Client is not installed. Installing the HDFS Client on all Storm worker nodes should resolve the problem.
