Repository: incubator-senssoft Updated Branches: refs/heads/master 87d7e09cb -> c3aa251df
Pushed huge update on SensSoft docker deployments. Upgraded to docker-compose version 3. Project: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/commit/c3aa251d Tree: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/tree/c3aa251d Diff: http://git-wip-us.apache.org/repos/asf/incubator-senssoft/diff/c3aa251d Branch: refs/heads/master Commit: c3aa251dfbee2753d0750de109fdf2284b1c8470 Parents: 87d7e09 Author: msbeard <msbe...@apache.org> Authored: Wed Nov 1 17:47:27 2017 -0400 Committer: msbeard <msbe...@apache.org> Committed: Wed Nov 1 17:47:27 2017 -0400 ---------------------------------------------------------------------- docker/README.md | 155 ++++++++++++++++---- docker/docker-compose.yml | 157 +++++++++++---------- docker/elasticsearch/Dockerfile | 27 ++++ docker/elasticsearch/config/elasticsearch.yml | 48 +++++++ docker/elasticsearch/config/log4j2.properties | 24 ++++ docker/es/elasticsearch.yml | 114 --------------- docker/images/configure_index.png | Bin 0 -> 116678 bytes docker/images/confirmation.png | Bin 0 -> 158876 bytes docker/images/dashboard.png | Bin 0 -> 172635 bytes docker/images/management.png | Bin 0 -> 60954 bytes docker/kibana/Dockerfile | 23 +++ docker/kibana/config/kibana.yml | 18 +++ docker/kibana/dashboard.json | 21 +-- docker/kibana/visualizations.json | 58 ++++++++ docker/logstash/Dockerfile | 9 +- docker/logstash/config/logstash-userale.conf | 2 +- docker/requirements.txt | 1 + docker/startup/entrypoint.sh | 3 +- 18 files changed, 426 insertions(+), 234 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/README.md ---------------------------------------------------------------------- diff --git a/docker/README.md b/docker/README.md index 2669bdb..f9d395a 100644 --- a/docker/README.md +++ b/docker/README.md @@ -1,35 +1,140 @@ -How to Build Docker Containers ------------------------------- +How to Build SensSoft Docker Containers +--------------------------------------- 1. Install [``Docker``](http://docker.com) on your machine. -2. Install ``docker-compose`` in an virtual environment. Full instructions can be found [``here``](https://docs.docker.com/compose/install/). - ``` - $ python3 -m venv env - $ source env/bin/activate - $ pip install docker-compose - ``` -3. Before launching the Docker containers, ensure your ``vm_max_map_count`` kernel setting is set to at least 262144. - Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions. + +1. Install ``docker-compose`` in an virtual environment. + Full instructions can be found [``here``](https://docs.docker.com/compose/install/). + + ```bash + $ python3 -m venv env + $ source env/bin/activate + $ pip install -r requirements.txt ``` - # For Linux systems + +1. Before launching the Docker containers, ensure your ``vm_max_map_count`` + kernel setting is set to at least 262144. + Visit [``Running Elasticsearch in Production mode``](https://www.elastic.co/guide/en/elasticsearch/reference/5.5/docker.html#docker-cli-run-prod-mode) for OS specific instructions. + + ```bash + # Example for Linux systems $ sysctl -w vm.max_map_count=262144 ``` -4. To build and run all ``Docker`` containers. - ``` - $ docker-compose up -d - ``` -5. To run a specific ``Docker`` container. - ``` - $ docker-compose up -d site - # Note: site container is instrumented w/ userale; all userale logs will be sent to the - # elasticsearch docker container. - ``` -6. Verify the deployment by navigating to ``Kibana`` in your favorite browser. - ```sh - http://localhost:5601 + +1. Start Elasticsearch cluster: + + ```bash + $ docker-compose up -d elasticsearch loadbalancer ``` + + The loadbalancer node exposes port 9200 on localhost and is the only node + that has HTTP enabled. Services such as Kibana and Logstash connect to the + loadbalancer node directly. Loadbalancer accepts requests from Kibana and Logstash + and balances them across the elasticsearch worker nodes. The elasticsearch + worker nodes communicate to each other and the loadbalancer via TCP on port 9300. + +1. Once Elasticsearch cluster is fully started, scale out elasticsearch service: + ```bash + $ docker-compose up --scale elasticsearch=3 + ``` + +1. Confirm cluster state: + ```bash + $ curl -XGET http://localhost:9200/_cluster/health?pretty + { + "cluster_name" : "SensSoft", + "status" : "green", + "timed_out" : false, + "number_of_nodes" : 4, + "number_of_data_nodes" : 3, + "active_primary_shards" : 0, + "active_shards" : 0, + "relocating_shards" : 0, + "initializing_shards" : 0, + "unassigned_shards" : 0, + "delayed_unassigned_shards" : 0, + "number_of_pending_tasks" : 0, + "number_of_in_flight_fetch" : 0, + "task_max_waiting_in_queue_millis" : 0, + "active_shards_percent_as_number" : 100.0 + } + ``` + Confirm that the `number_of_nodes` is 4 and `number_of_data_nodes` is 3. + +1. Launch logging server. Give Logstash about 2 minutes to start before confirming + its state. + + ```bash + $ docker-compose up -d logstash + + $ curl -XGET http://localhost:8100 + ok + ``` + +1. Before Kibana can be used, we will need to generate some data. We have already + provided an example instrumented website to assist. + + ```bash + $ docker-compose up -d site + ``` + + Visit `http://localhost:8080` and you will see Apache SensSoft's home page. + +1. Launch Kibana. Give Kibana about 2-5 minutes to start before accessing + `http://localhost:5601`. + + ```bash + $ docker-compose up -d kibana + ``` + +1. Register an index in Kibana to see the logs: + + Goto: Management -> Index Patterns and enter `userale` in the Index pattern box. + Choose `clientTime` in the drop down `Time Filter field name` field. + + ![alt text][configure_index] + +1. Load example Dashboard and Visualizations under docker/kibana/. + + Goto: Management -> Saved Objects and select the `Import` button. Import the + `visualizations.json` and `dashboard.json` file. + + ![alt text][management] -7. Stop all the containers. + Confirm index conflicts if message appears. + + ![alt text][confirmation] + + Once that is complete, navigate to the `Dashboard` view in Kibana and click the + `Apache SensSoft Dashboard` object. + + ![alt text][dashboard] + +1. To stop all containers. ```sh $ docker-compose stop ``` + +Having Issues? +-------------- +1. Check out the docker-compose logs for the service(s) that are having issues. + + ```bash + $ docker-compose ps + ``` +1. Make sure to send us the docker-compose logs to help diagnose your issues please! + + ```bash + $ docker-compose logs > err.dump + ``` + +Todo +---- +- [ ] TAP docker deployment instructions. +- [ ] Distill docker deployment instructions. +- [ ] Apache SensSoft Docker + Kubernetes instructions. + +[configure_index]: ./images/configure_index.png "Configure Kibana index" +[confirmation]: ./images/confirmation.png "Confirm index pattern conflicts" +[dashboard]: ./images/dashboard.png "Apache Senssoft Dashboard" +[management]: ./images/management.png "Kibana management console" \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/docker-compose.yml ---------------------------------------------------------------------- diff --git a/docker/docker-compose.yml b/docker/docker-compose.yml index fc14c1c..17b1106 100644 --- a/docker/docker-compose.yml +++ b/docker/docker-compose.yml @@ -15,131 +15,140 @@ # Apache SensSoft Docker Build -version: "2" +version: "3" services: # Startup startup: build: ./startup + image: senssoft.incubator.apache.org/startup:latest container_name: senssoft-start depends_on: - - "elasticsearch1" - - "elasticsearch2" + - "loadbalancer" command: bash /opt/entrypoint.sh # Site site: build: ./site + image: senssoft.incubator.apache.org/website:latest container_name: senssoft-app ports: - 8080:8080 depends_on: - - "userale-forwarder" + - "logstash" command: python -m SimpleHTTPServer 8080 - # networks: - # - esnet - # ELK Stack - - # The node elasticsearch1 listens on localhost:9200 while elasticsearch2 - # talks to elasticsearch1 over a Docker network. - elasticsearch1: - image: docker.elastic.co/elasticsearch/elasticsearch:5.5.0 - container_name: senssoft-elastic-1 + # Elasticsearch Workers + elasticsearch: + build: ./elasticsearch + image: senssoft.incubator.apache.org/elasticsearch:5.6.3 + command: + - /bin/sh + - -c + - | + elasticsearch-plugin remove --silent x-pack + elasticsearch environment: - - cluster.name=SensSoft - - node.name=soft-01 - - bootstrap.memory_lock=true - - xpack.security.enabled=false + - NODE_MASTER=true + - NUMBER_OF_MASTERS=2 + - NODE_COORDINATE=true + - "DISCOVERY_SERVICE=elasticsearch" + - NODE_DATA=true + - NODE_INGEST=true + - MAX_LOCAL_STORAGE_NODES=1 + - NETWORK_HOST=0.0.0.0 + - HTTP_ENABLE=false + - HTTP_CORS_ENABLE=false + - HTTP_CORS_ALLOW_ORIGIN=* - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - ulimits: - memlock: - soft: -1 - hard: -1 - mem_limit: 1g - volumes: - - esdata1:/usr/share/elasticsearch/data - ports: - - 9200:9200 - # networks: - # - esnet + networks: + - esnet - elasticsearch2: - image: docker.elastic.co/elasticsearch/elasticsearch:5.5.0 - container_name: senssoft-elastic-2 + # Elasticsearch Loadbalancer + loadbalancer: + build: ./elasticsearch + image: senssoft.incubator.apache.org/elasticsearch:5.6.3 + container_name: senssoft-loadbalancer + command: + - /bin/sh + - -c + - | + elasticsearch-plugin remove --silent x-pack + elasticsearch environment: - - cluster.name=SensSoft - - node.name=soft-02 - - bootstrap.memory_lock=true - - xpack.security.enabled=false + - NODE_MASTER=false + - NUMBER_OF_MASTERS=1 + - NODE_COORDINATE=true + - "DISCOVERY_SERVICE=elasticsearch" + - NODE_DATA=false + - NODE_INGEST=false + - MAX_LOCAL_STORAGE_NODES=1 + - NETWORK_HOST=0.0.0.0 + - HTTP_ENABLE=true + - HTTP_CORS_ENABLE=true + - HTTP_CORS_ALLOW_ORIGIN=* - "ES_JAVA_OPTS=-Xms512m -Xmx512m" - - "discovery.zen.ping.unicast.hosts=elasticsearch1" - ulimits: - memlock: - soft: -1 - hard: -1 - mem_limit: 1g - volumes: - - esdata2:/usr/share/elasticsearch/data - # networks: - # - esnet + ports: + - "9200:9200" + - "9300" + networks: + - esnet # Kibana kibana: - image: docker.elastic.co/kibana/kibana:5.5.0 + build: ./kibana + image: senssoft.incubator.apache.org/kibana:5.6.3 container_name: senssoft-kibana + environment: + - "ES_JAVA_OPTS=-Xms2g -Xmx2g" ports: - - 5601:5601 + - "5601:5601" depends_on: - - "elasticsearch1" - - "elasticsearch2" - - "startup" - environment: - XPACK_MONITORING_ENABLED: "false" - ELASTICSEARCH_URL: http://elasticsearch1:9200 - # networks: - # - esnet + - "loadbalancer" + networks: + - esnet # Logstash - userale-forwarder: + logstash: build: ./logstash - container_name: senssoft-userale-forwarder + image: senssoft.incubator.apache.org/logstash:5.6.3 + container_name: senssoft-logstash command: -f /etc/logstash/conf.d ports: - 8100:8100 - volumes: + volumes: - ./logstash/config/logstash-userale.conf:/etc/logstash/conf.d/logstash-userale.conf - ./logstash/templates/userale.json:/usr/share/logstash/templates/userale.json depends_on: - - "elasticsearch1" - - "elasticsearch2" - environment: - XPACK_MONITORING_ENABLED: "false" - # networks: - # - esnet + - "loadbalancer" + networks: + - esnet # Distill distill: build: ./distill + image: senssoft.incubator.apache.org/distill:0.1.0 container_name: senssoft-distill ports: - 8090:8090 depends_on: - - "elasticsearch1" - # networks: - # - esnet + - "loadbalancer" + networks: + - esnet # TAP's Database db: - container_name: senssoft-db build: ./db + image: senssoft.incubator.apache.org/postgres:latest + container_name: senssoft-postgres ports: - "5432:5432" # TAP tap: - container_name: senssoft-tap build: ./tap + image: senssoft.incubator.apache.org/tap:0.1.0 + container_name: senssoft-tap # command: ["bash", "./wait-for-postgres.sh"] command: bash -c "/usr/src/app/wait-for-postgres.sh" ports: @@ -149,11 +158,7 @@ services: links: - db:db -volumes: - esdata1: - driver: local - esdata2: - driver: local +networks: -# networks: -# esnet: + esnet: + driver: bridge http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/Dockerfile ---------------------------------------------------------------------- diff --git a/docker/elasticsearch/Dockerfile b/docker/elasticsearch/Dockerfile new file mode 100644 index 0000000..7cdbae7 --- /dev/null +++ b/docker/elasticsearch/Dockerfile @@ -0,0 +1,27 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM docker.elastic.co/elasticsearch/elasticsearch:5.6.3 + +MAINTAINER Michelle Beard <msbe...@apache.org> + +# Override config +COPY config /usr/share/elasticsearch/config + +# Set environment +ENV CLUSTER_NAME SensSoft + +# Kubernetes requires swap is turned off +ENV MEMORY_LOCK false http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/config/elasticsearch.yml ---------------------------------------------------------------------- diff --git a/docker/elasticsearch/config/elasticsearch.yml b/docker/elasticsearch/config/elasticsearch.yml new file mode 100644 index 0000000..ca31f63 --- /dev/null +++ b/docker/elasticsearch/config/elasticsearch.yml @@ -0,0 +1,48 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +cluster: + name: ${CLUSTER_NAME} + +node: + master: ${NODE_MASTER} + data: ${NODE_DATA} + #name: ${NODE_NAME} + ingest: ${NODE_INGEST} + max_local_storage_nodes: ${MAX_LOCAL_STORAGE_NODES} + +network.host: ${NETWORK_HOST} + +search.remote.connect: ${NODE_COORDINATE} + +path: + data: /usr/share/elasticsearch/data + logs: /usr/share/elasticsearch/log + +bootstrap: + memory_lock: ${MEMORY_LOCK} + +http: + enabled: ${HTTP_ENABLE} + compression: true + cors: + enabled: ${HTTP_CORS_ENABLE} + allow-origin: ${HTTP_CORS_ALLOW_ORIGIN} + +# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1): +discovery: + zen: + ping.unicast.hosts: ${DISCOVERY_SERVICE} + minimum_master_nodes: ${NUMBER_OF_MASTERS} http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/elasticsearch/config/log4j2.properties ---------------------------------------------------------------------- diff --git a/docker/elasticsearch/config/log4j2.properties b/docker/elasticsearch/config/log4j2.properties new file mode 100644 index 0000000..7f7817d --- /dev/null +++ b/docker/elasticsearch/config/log4j2.properties @@ -0,0 +1,24 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +status = error + +appender.console.type = Console +appender.console.name = console +appender.console.layout.type = PatternLayout +appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n + +rootLogger.level = info +rootLogger.appenderRef.console.ref = console \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/es/elasticsearch.yml ---------------------------------------------------------------------- diff --git a/docker/es/elasticsearch.yml b/docker/es/elasticsearch.yml deleted file mode 100644 index 5e0c014..0000000 --- a/docker/es/elasticsearch.yml +++ /dev/null @@ -1,114 +0,0 @@ -# Licensed to the Apache Software Foundation (ASF) under one or more -# contributor license agreements. See the NOTICE file distributed with -# this work for additional information regarding copyright ownership. -# The ASF licenses this file to You under the Apache License, Version 2.0 -# (the "License"); you may not use this file except in compliance with -# the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -# Unless required by applicable law or agreed to in writing, software -# distributed under the License is distributed on an "AS IS" BASIS, -# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. -# See the License for the specific language governing permissions and -# limitations under the License. - -# ======================== Elasticsearch Configuration ========================= -# -# NOTE: Elasticsearch comes with reasonable defaults for most settings. -# Before you set out to tweak and tune the configuration, make sure you -# understand what are you trying to accomplish and the consequences. -# -# The primary way of configuring a node is via this file. This template lists -# the most important settings you may want to configure for a production cluster. -# -# Please see the documentation for further information on configuration options: -# <http://www.elastic.co/guide/en/elasticsearch/reference/current/setup-configuration.html> -# -# ---------------------------------- Cluster ----------------------------------- -# -# Use a descriptive name for your cluster: -# -cluster.name: SensSoft -# -# ------------------------------------ Node ------------------------------------ -# -# Use a descriptive name for the node: -# -node.name: soft-01 -# -# Add custom attributes to the node: -# -# node.rack: r1 -# -# ----------------------------------- Paths ------------------------------------ -# -# Path to directory where to store the data (separate multiple locations by comma): -# -path.data: /usr/share/elasticsearch/data -# -# Path to log files: -# -path.logs: /usr/share/elasticsearch/logs -# -# -# ----------------------------------- Memory ----------------------------------- -# -# Lock the memory on startup: -# -# bootstrap.mlockall: true -# -# Make sure that the `ES_HEAP_SIZE` environment variable is set to about half the memory -# available on the system and that the owner of the process is allowed to use this limit. -# -# Elasticsearch performs poorly when the system is swapping the memory. -# -# ---------------------------------- Network ----------------------------------- -# -# Set the bind address to a specific IP (IPv4 or IPv6): -# -network.host: 0.0.0.0 -# -# Set a custom port for HTTP: -# -# http.port: 9200 -# -# For more information, see the documentation at: -# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html> -# -# --------------------------------- Discovery ---------------------------------- -# -# Pass an initial list of hosts to perform discovery when new node is started: -# The default list of hosts is ["127.0.0.1", "[::1]"] -# -# discovery.zen.ping.unicast.hosts: ["host1", "host2"] -# -# Prevent the "split brain" by configuring the majority of nodes (total number of nodes / 2 + 1): -# -discovery.zen.minimum_master_nodes: 1 -# -# For more information, see the documentation at: -# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery.html> -# -# ---------------------------------- Gateway ----------------------------------- -# -# Block initial recovery after a full cluster restart until N nodes are started: -# -# gateway.recover_after_nodes: 3 -# -# For more information, see the documentation at: -# <http://www.elastic.co/guide/en/elasticsearch/reference/current/modules-gateway.html> -# -# ---------------------------------- Various ----------------------------------- -# -# Disable starting multiple nodes on a single system: -# -# node.max_local_storage_nodes: 1 -# -# Require explicit names when deleting indices: -# -# action.destructive_requires_name: true - -http.cors.enabled: true -http.cors.allow-origin: "*" -xpack.security.enabled: false \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/configure_index.png ---------------------------------------------------------------------- diff --git a/docker/images/configure_index.png b/docker/images/configure_index.png new file mode 100644 index 0000000..31ebe02 Binary files /dev/null and b/docker/images/configure_index.png differ http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/confirmation.png ---------------------------------------------------------------------- diff --git a/docker/images/confirmation.png b/docker/images/confirmation.png new file mode 100644 index 0000000..467b4d4 Binary files /dev/null and b/docker/images/confirmation.png differ http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/dashboard.png ---------------------------------------------------------------------- diff --git a/docker/images/dashboard.png b/docker/images/dashboard.png new file mode 100644 index 0000000..7123c9e Binary files /dev/null and b/docker/images/dashboard.png differ http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/images/management.png ---------------------------------------------------------------------- diff --git a/docker/images/management.png b/docker/images/management.png new file mode 100644 index 0000000..90a26ae Binary files /dev/null and b/docker/images/management.png differ http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/Dockerfile ---------------------------------------------------------------------- diff --git a/docker/kibana/Dockerfile b/docker/kibana/Dockerfile new file mode 100644 index 0000000..894fd53 --- /dev/null +++ b/docker/kibana/Dockerfile @@ -0,0 +1,23 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +FROM docker.elastic.co/kibana/kibana:5.6.3 + +MAINTAINER Michelle Beard <msbe...@apache.org> + +# Override config +COPY config /usr/share/kibana/config + +RUN kibana-plugin remove x-pack http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/config/kibana.yml ---------------------------------------------------------------------- diff --git a/docker/kibana/config/kibana.yml b/docker/kibana/config/kibana.yml new file mode 100644 index 0000000..20f2999 --- /dev/null +++ b/docker/kibana/config/kibana.yml @@ -0,0 +1,18 @@ +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +server.name: kibana +server.host: "0" +elasticsearch.url: http://loadbalancer:9200 http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/dashboard.json ---------------------------------------------------------------------- diff --git a/docker/kibana/dashboard.json b/docker/kibana/dashboard.json index d1922d9..8a179a6 100644 --- a/docker/kibana/dashboard.json +++ b/docker/kibana/dashboard.json @@ -1,25 +1,18 @@ [ { - "_id": "6e6b6380-3673-11e7-b2f7-0b10e220d9ac", + "_id": "513c8680-be7d-11e7-8adf-fd0a12531eca", "_type": "dashboard", "_source": { - "title": "Apache UserALE Monitor", + "title": "Apache SensSoft Dashboard", "hits": 0, - "description": "Example Kibana Dashboard", - "panelsJSON": "[\n {\n \"size_x\": 9,\n \"size_y\": 4,\n \"panelIndex\": 1,\n \"type\": \"visualization\",\n \"id\": \"49fa7860-3673-11e7-b2f7-0b10e220d9ac\",\n \"col\": 4,\n \"row\": 4\n },\n {\n \"size_x\": 3,\n \"size_y\": 3,\n \"panelIndex\": 2,\n \"type\": \"visualization\",\n \"id\": \"6e55ca90-3671-11e7-b2f7-0b10e220d9ac\",\n \"col\": 1,\n \"row\": 4\n },\n {\n \"size_x\": 5,\n \"size_y\": 3,\n \"panelIndex\": 3,\n \"type\": \"visualization\",\n \"id\": \"ef62c4e0-3684-11e7-9c77-0135be018b90\",\n \"col\": 1,\n \"row\": 1\n },\n {\n \"size_x\": 7,\n \"size_y\": 3,\n \"panelIndex\": 4,\n \"type\": \"visualization\",\n \"id\": \"7ee8abd0-3684-11e7-9c77-0135be018b90\",\n \"col\": 6,\n \"row\": 1\n }\n]", - "optionsJSON": "{\n \"darkTheme\": false\n}", + "description": "", + "panelsJSON": "[{\"size_x\":6,\"size_y\":5,\"panelIndex\":1,\"type\":\"visualization\",\"id\":\"2bf40010-be7d-11e7-8adf-fd0a12531eca\",\"col\":1,\"row\":4},{\"size_x\":6,\"size_y\":3,\"panelIndex\":2,\"type\":\"visualization\",\"id\":\"f7ca4560-be7c-11e7-8adf-fd0a12531eca\",\"col\":7,\"row\":1},{\"size_x\":6,\"size_y\":3,\"panelIndex\":3,\"type\":\"visualization\",\"id\":\"bcd7c950-be7c-11e7-8adf-fd0a12531eca\",\"col\":7,\"row\":4},{\"size_x\":6,\"size_y\":3,\"panelIndex\":4,\"type\":\"visualization\",\"id\":\"e36b1f40-be7c-11e7-8adf-fd0a12531eca\",\"col\":1,\"row\":1}]", + "optionsJSON": "{\"darkTheme\":true}", "uiStateJSON": "{}", "version": 1, - "timeRestore": true, - "timeTo": "now", - "timeFrom": "now-15m", - "refreshInterval": { - "display": "Off", - "pause": false, - "value": 0 - }, + "timeRestore": false, "kibanaSavedObjectMeta": { - "searchSourceJSON": "{\n \"filter\": [\n {\n \"query\": {\n \"query_string\": {\n \"query\": \"*\",\n \"analyze_wildcard\": true\n }\n }\n }\n ],\n \"highlightAll\": true,\n \"version\": true\n}" + "searchSourceJSON": "{\"filter\":[{\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}}}],\"highlightAll\":true,\"version\":true}" } } } http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/kibana/visualizations.json ---------------------------------------------------------------------- diff --git a/docker/kibana/visualizations.json b/docker/kibana/visualizations.json new file mode 100644 index 0000000..7052384 --- /dev/null +++ b/docker/kibana/visualizations.json @@ -0,0 +1,58 @@ +[ + { + "_id": "bcd7c950-be7c-11e7-8adf-fd0a12531eca", + "_type": "visualization", + "_source": { + "title": "Top Event Types", + "visState": "{\"title\":\"Top Event Types\",\"type\":\"pie\",\"params\":{\"addTooltip\":true,\"addLegend\":true,\"legendPosition\":\"right\",\"isDonut\":false},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Top Event Types\"}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"type\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"Event types\"}}],\"listeners\":{}}", + "uiStateJSON": "{}", + "description": "", + "version": 1, + "kibanaSavedObjectMeta": { + "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" + } + } + }, + { + "_id": "e36b1f40-be7c-11e7-8adf-fd0a12531eca", + "_type": "visualization", + "_source": { + "title": "Total Number of Logs", + "visState": "{\"title\":\"Total Number of Logs\",\"type\":\"metric\",\"params\":{\"handleNoResults\":true,\"fontSize\":60},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{\"customLabel\":\"Total number of logs\"}}],\"listeners\":{}}", + "uiStateJSON": "{}", + "description": "", + "version": 1, + "kibanaSavedObjectMeta": { + "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" + } + } + }, + { + "_id": "f7ca4560-be7c-11e7-8adf-fd0a12531eca", + "_type": "visualization", + "_source": { + "title": "Time Series", + "visState": "{\"title\":\"Time Series\",\"type\":\"timelion\",\"params\":{\"expression\":\".es(*)\",\"interval\":\"auto\"},\"aggs\":[],\"listeners\":{}}", + "uiStateJSON": "{}", + "description": "", + "version": 1, + "kibanaSavedObjectMeta": { + "searchSourceJSON": "{\"query\":{\"query_string\":{\"query\":\"*\"}},\"filter\":[]}" + } + } + }, + { + "_id": "2bf40010-be7d-11e7-8adf-fd0a12531eca", + "_type": "visualization", + "_source": { + "title": "Events by Targets", + "visState": "{\"title\":\"Events by Targets\",\"type\":\"histogram\",\"params\":{\"addLegend\":true,\"addTimeMarker\":false,\"addTooltip\":true,\"categoryAxes\":[{\"id\":\"CategoryAxis-1\",\"labels\":{\"show\":true,\"truncate\":100},\"position\":\"bottom\",\"scale\":{\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{\"text\":\"HTML Tags\"},\"type\":\"category\"}],\"defaultYExtents\":false,\"drawLinesBetweenPoints\":true,\"grid\":{\"categoryLines\":false,\"style\":{\"color\":\"#eee\"}},\"interpolate\":\"linear\",\"legendPosition\":\"right\",\"radiusRatio\":9,\"scale\":\"linear\",\"seriesParams\":[{\"data\":{\"id\":\"1\",\"label\":\"Count\"},\"drawLinesBetweenPoints\":true,\"mode\":\"stacked\",\"show\":\"true\",\"showCircles\":true,\"type\":\"histogram\",\"valueAxis\":\"ValueAxis-1\"}],\"setYExtents\":false,\"showCircles\":true,\"times\":[],\"valueAxes\":[{\"id\":\"ValueAxis-1\",\"labels\":{\"filter\":false,\"rotate\":0,\"show\":true,\"truncate\":100},\"name\":\"LeftAxi s-1\",\"position\":\"left\",\"scale\":{\"mode\":\"normal\",\"type\":\"linear\"},\"show\":true,\"style\":{},\"title\":{},\"type\":\"value\"}]},\"aggs\":[{\"id\":\"1\",\"enabled\":true,\"type\":\"count\",\"schema\":\"metric\",\"params\":{}},{\"id\":\"2\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"segment\",\"params\":{\"field\":\"target\",\"size\":100,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"HTML Tags\"}},{\"id\":\"3\",\"enabled\":true,\"type\":\"terms\",\"schema\":\"group\",\"params\":{\"field\":\"type\",\"size\":20,\"order\":\"desc\",\"orderBy\":\"1\",\"customLabel\":\"JS Events\"}}],\"listeners\":{}}", + "uiStateJSON": "{}", + "description": "", + "version": 1, + "kibanaSavedObjectMeta": { + "searchSourceJSON": "{\"index\":\"userale\",\"query\":{\"query_string\":{\"query\":\"*\",\"analyze_wildcard\":true}},\"filter\":[]}" + } + } + } +] \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/logstash/Dockerfile ---------------------------------------------------------------------- diff --git a/docker/logstash/Dockerfile b/docker/logstash/Dockerfile index 776f935..6aebb9e 100644 --- a/docker/logstash/Dockerfile +++ b/docker/logstash/Dockerfile @@ -13,7 +13,12 @@ # See the License for the specific language governing permissions and # limitations under the License. -FROM docker.elastic.co/logstash/logstash:5.5.0 +FROM docker.elastic.co/logstash/logstash:5.6.3 MAINTAINER Michelle Beard <msbe...@apache.org> -RUN mkdir /usr/share/logstash/templates \ No newline at end of file +RUN mkdir /usr/share/logstash/templates + +RUN logstash-plugin remove x-pack + +# Known issue: https://github.com/elastic/logstash-docker/issues/45 +RUN sed -i '/xpack/d' /usr/share/logstash/config/logstash.yml http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/logstash/config/logstash-userale.conf ---------------------------------------------------------------------- diff --git a/docker/logstash/config/logstash-userale.conf b/docker/logstash/config/logstash-userale.conf index a16c434..cbe8b8c 100644 --- a/docker/logstash/config/logstash-userale.conf +++ b/docker/logstash/config/logstash-userale.conf @@ -39,7 +39,7 @@ filter { output { # Output data to Elasticsearch instance elasticsearch { - hosts => "elasticsearch1:9200" + hosts => "loadbalancer:9200" index => "userale" document_type => "logs" manage_template => true http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/requirements.txt ---------------------------------------------------------------------- diff --git a/docker/requirements.txt b/docker/requirements.txt new file mode 100644 index 0000000..6094def --- /dev/null +++ b/docker/requirements.txt @@ -0,0 +1 @@ +docker-compose==1.16.1 http://git-wip-us.apache.org/repos/asf/incubator-senssoft/blob/c3aa251d/docker/startup/entrypoint.sh ---------------------------------------------------------------------- diff --git a/docker/startup/entrypoint.sh b/docker/startup/entrypoint.sh index 00953e6..c33a63b 100644 --- a/docker/startup/entrypoint.sh +++ b/docker/startup/entrypoint.sh @@ -17,5 +17,4 @@ # Wait for the Elasticsearch container to be ready before starting Kibana. echo "Stalling for Elasticsearch" -dockerize -wait http://elasticsearch1:9200 -dockerize -wait http://elasticsearch2:9200 \ No newline at end of file +dockerize -wait http://loadbalancer:9200
