Repository: incubator-griffin Updated Branches: refs/heads/master 509640f7e -> 8fffb4732
Docker readme and guide document Author: Lionel Liu <[email protected]> Closes #160 from bhlx3lyx7/docker. Project: http://git-wip-us.apache.org/repos/asf/incubator-griffin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-griffin/commit/8fffb473 Tree: http://git-wip-us.apache.org/repos/asf/incubator-griffin/tree/8fffb473 Diff: http://git-wip-us.apache.org/repos/asf/incubator-griffin/diff/8fffb473 Branch: refs/heads/master Commit: 8fffb4732e89c0d0fd31551684a3f8480e39e238 Parents: 509640f Author: Lionel Liu <[email protected]> Authored: Thu Nov 2 14:38:41 2017 +0800 Committer: Lionel Liu <[email protected]> Committed: Thu Nov 2 14:38:41 2017 +0800 ---------------------------------------------------------------------- README.md | 15 ++- .../docker/svc_msr/docker-compose-batch.yml | 42 ++++++++ .../docker/svc_msr/docker-compose-streaming.yml | 61 +++++++++++ griffin-doc/docker/svc_msr/docker-compose.yml | 42 -------- griffin-doc/griffin-docker-guide.md | 105 +++++++++++++++++++ 5 files changed, 219 insertions(+), 46 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/8fffb473/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 872c367..452b3f2 100644 --- a/README.md +++ b/README.md @@ -40,23 +40,30 @@ Release: ### How to run in docker 1. Install [docker](https://docs.docker.com/engine/installation/) and [docker compose](https://docs.docker.com/compose/install/). -2. Pull our built docker image and elasticsearch image. +2. Pull our pre-built docker image and elasticsearch image. ``` docker pull bhlx3lyx7/svc_msr:0.1.6 - docker pull elasticsearch:5 + docker pull bhlx3lyx7/elasticsearch + ``` + You can pull the images faster through mirror acceleration if you are in China. + ``` + docker pull registry.docker-cn.com/bhlx3lyx7/svc_msr:0.1.6 + docker pull registry.docker-cn.com/bhlx3lyx7/elasticsearch ``` 3. Increase vm.max_map_count of your local machine, to use elasticsearch. ``` sysctl -w vm.max_map_count=262144 ``` -4. Copy [docker-compose.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose.yml) to your work path. +4. Copy [docker-compose-batch.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-batch.yml) to your work path. 5. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. ``` - docker-compose up -d + docker-compose -f docker-compose-batch.yml up -d ``` 6. Now you can try griffin APIs by using postman after importing the [json files](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/postman). In which you need to modify the environment `BASE_PATH` value into `<your local IP address>:38080`. +More details about griffin docker [here](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/griffin-docker-guide.md). + ### How to deploy and run at local 1. Install jdk (1.8 or later versions). 2. Install mysql. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/8fffb473/griffin-doc/docker/svc_msr/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-batch.yml b/griffin-doc/docker/svc_msr/docker-compose-batch.yml new file mode 100644 index 0000000..f542247 --- /dev/null +++ b/griffin-doc/docker/svc_msr/docker-compose-batch.yml @@ -0,0 +1,42 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/svc_msr:0.1.6 + hostname: griffin + links: + - es + environment: + ES_HOSTNAME: es + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + - 39300:9300 + container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/8fffb473/griffin-doc/docker/svc_msr/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml new file mode 100644 index 0000000..9fde137 --- /dev/null +++ b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml @@ -0,0 +1,61 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/svc_msr:0.1.6 + hostname: griffin + links: + - es + - zk + - kafka + environment: + ES_HOSTNAME: es + ZK_HOSTNAME: zk + KAFKA_HOSTNAME: kafka + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + container_name: es + +zk: + image: zookeeper:3.5 + hostname: zk + ports: + - 32181:2181 + container_name: zk + restart: always + +kafka: + image: bhlx3lyx7/kafka + hostname: kafka + ports: + - 39092:9092 + container_name: kafka + tty: true \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/8fffb473/griffin-doc/docker/svc_msr/docker-compose.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose.yml b/griffin-doc/docker/svc_msr/docker-compose.yml deleted file mode 100644 index a9e1571..0000000 --- a/griffin-doc/docker/svc_msr/docker-compose.yml +++ /dev/null @@ -1,42 +0,0 @@ -#Licensed to the Apache Software Foundation (ASF) under one -#or more contributor license agreements. See the NOTICE file -#distributed with this work for additional information -#regarding copyright ownership. The ASF licenses this file -#to you under the Apache License, Version 2.0 (the -#"License"); you may not use this file except in compliance -#with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -#Unless required by applicable law or agreed to in writing, -#software distributed under the License is distributed on an -#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -#KIND, either express or implied. See the License for the -#specific language governing permissions and limitations -#under the License. - -griffin: - image: bhlx3lyx7/svc_msr:0.1.6 - hostname: griffin - links: - - es - environment: - ES_HOSTNAME: es - ports: - - 32122:2122 - - 38088:8088 - - 33306:3306 - - 38042:8042 - - 39083:9083 - - 38998:8998 - - 38080:8080 - tty: true - container_name: griffin - -es: - image: elasticsearch:5 - hostname: es - ports: - - 39200:9200 - - 39300:9300 - container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/8fffb473/griffin-doc/griffin-docker-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/griffin-docker-guide.md b/griffin-doc/griffin-docker-guide.md new file mode 100644 index 0000000..1fb5980 --- /dev/null +++ b/griffin-doc/griffin-docker-guide.md @@ -0,0 +1,105 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Apache Griffin Docker Guide +Griffin docker images are pre-built on docker hub, users can pull them to try griffin in docker. + +## Preparation + +### Environment preparation +1. Install [docker](https://docs.docker.com/engine/installation/) and [docker compose](https://docs.docker.com/compose/install/). +2. Increase vm.max_map_count of your local machine, to use elasticsearch. + ``` + sysctl -w vm.max_map_count=262144 + ``` +3. Pull griffin pre-built docker images. + ``` + docker pull bhlx3lyx7/svc_msr:0.1.6 + docker pull bhlx3lyx7/elasticsearch + docker pull bhlx3lyx7/kafka + docker pull zookeeper:3.5 + ``` + Or you can pull the images faster through mirror acceleration if you are in China. + ``` + docker pull registry.docker-cn.com/bhlx3lyx7/svc_msr:0.1.6 + docker pull registry.docker-cn.com/bhlx3lyx7/elasticsearch + docker pull registry.docker-cn.com/bhlx3lyx7/kafka + docker pull registry.docker-cn.com/zookeeper:3.5 + ``` + The docker images are the griffin environment images. + - `bhlx3lyx7/svc_msr`: This image contains mysql, hadoop, hive, spark, livy, griffin service, griffin measure, and some prepared demo data, it works as a single node spark cluster, providing spark engine and griffin service. + - `bhlx3lyx7/elasticsearch`: This image is based on official elasticsearch, adding some configurations to enable cors requests, to provide elasticsearch service for metrics persist. + - `bhlx3lyx7/kafka`: This image contains kafka 0.8, and some demo streaming data, to provide streaming data source in streaming mode. + - `zookeeper:3.5`: This image is official zookeeper, to provide zookeeper service in streaming mode. + +### How to use griffin docker images in batch mode +1. Copy [docker-compose-batch.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-batch.yml) to your work path. +2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. + ``` + docker-compose -f docker-compose-batch.yml up -d + ``` +3. Now you can try griffin APIs by using postman after importing the [json files](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/postman). + In which you need to modify the environment `BASE_PATH` value into `<your local IP address>:38080`. +4. You can try the api `Basic -> Get griffin version`, to make sure griffin service has started up. +5. Add an accuracy measure through api `Measures -> Add measure`, to create a measure in griffin. +6. Add a job to through api `jobs -> Add job`, to schedule a job to execute the measure. In the example, the schedule interval is 5 minutes. +7. After some minutes, you can get the metrics from elasticsearch. + ``` + curl -XGET '<your local IP address>:39200/griffin/accuracy/_search?pretty&filter_path=hits.hits._source' -d '{"query":{"match_all":{}}, "sort": [{"tmst": {"order": "asc"}}]}' + ``` + +### How to use griffin docker images in streaming mode +1. Copy [docker-compose-streaming.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-streaming.yml) to your work path. +2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. + ``` + docker-compose -f docker-compose-streaming.yml up -d + ``` +3. Enter the griffin docker container. + ``` + docker exec -it griffin bash + ``` +4. Switch into the measure directory. + ``` + cd ~/measure + ``` +5. Execute the script of streaming-accu, to execute streaming accuracy measurement. + ``` + ./streaming-accu.sh + ``` + You can trace the log in streaming-accu.log. + ``` + tail -f streaming-accu.log + ``` +6. Limited by the docker container resource, you can only execute accuracy or profiling separately. + If you want to try streaming profiling measurement, please kill the streaming-accu process first. + ``` + kill -9 `ps -ef | awk '/griffin-measure/{print $2}'` + ``` + Then clear the checkpoint directory and other related directories of last streaming job. + ``` + ./clear.sh + ``` + Execute the script of streaming-prof, to execute streaming profiling measurement. + ``` + ./streaming-prof.sh + ``` + You can trace the log in streaming-prof.log. + ``` + tail -f streaming-prof.log + ``` \ No newline at end of file
