Repository: incubator-griffin Updated Branches: refs/heads/master 5fc8e8999 -> 293217f7a
Document of dev env build based on new built docker image with postgresql Author: Lionel Liu <[email protected]> Closes #243 from bhlx3lyx7/tmst. Project: http://git-wip-us.apache.org/repos/asf/incubator-griffin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-griffin/commit/293217f7 Tree: http://git-wip-us.apache.org/repos/asf/incubator-griffin/tree/293217f7 Diff: http://git-wip-us.apache.org/repos/asf/incubator-griffin/diff/293217f7 Branch: refs/heads/master Commit: 293217f7af32eee8075fd0460a969b2b56d503ba Parents: 5fc8e89 Author: Lionel Liu <[email protected]> Authored: Thu Mar 29 14:26:00 2018 +0800 Committer: Lionel Liu <[email protected]> Committed: Thu Mar 29 14:26:00 2018 +0800 ---------------------------------------------------------------------- README.md | 16 ++- griffin-doc/dev/dev-env-build.md | 144 +++++++++++++++++++ .../docker/svc_msr/docker-compose-batch.yml | 1 + .../docker/svc_msr/docker-compose-streaming.yml | 1 + 4 files changed, 159 insertions(+), 3 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/293217f7/README.md ---------------------------------------------------------------------- diff --git a/README.md b/README.md index 90a747c..5bc0e1c 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,7 @@ To run Griffin at local, you can follow instructions below. You need to install following items - jdk (1.8 or later versions). - mysql. +- Postgresql. - npm (version 6.0.0+). - [Hadoop](http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz) (2.6.0 or later), you can get some help [here](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html). - [Spark](http://spark.apache.org/downloads.html) (version 1.6.x, griffin does not support 2.0.x at current), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help [here](http://why-not-learn-something.blogspot.com/2015/06/spark-installation-pseudo.html). @@ -69,12 +70,15 @@ You should also modify some configurations of Griffin for your environment. - <b>service/src/main/resources/application.properties</b> ``` - # mysql - spring.datasource.url = jdbc:mysql://<your IP>:3306/quartz?autoReconnect=true&useSSL=false + # jpa + spring.datasource.url = jdbc:postgresql://<your IP>:5432/quartz?autoReconnect=true&useSSL=false spring.datasource.username = <user name> spring.datasource.password = <password> + spring.jpa.generate-ddl=true + spring.datasource.driverClassName = org.postgresql.Driver + spring.jpa.show-sql = true - # hive + # hive metastore hive.metastore.uris = thrift://<your IP>:9083 hive.metastore.dbname = <hive database name> # default is "default" @@ -167,6 +171,12 @@ You can use UI following the steps [here](https://github.com/apache/incubator-g **Note**: The front-end UI is still under development, you can only access some basic features currently. + +### Build and Debug + +If you want to develop Griffin, please follow [this document](griffin-doc/dev/dev-env-build.md), to skip complex environment building work. + + ## Community You can contact us via email: <a href="mailto:[email protected]";>[email protected]</a> http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/293217f7/griffin-doc/dev/dev-env-build.md ---------------------------------------------------------------------- diff --git a/griffin-doc/dev/dev-env-build.md b/griffin-doc/dev/dev-env-build.md new file mode 100644 index 0000000..6d842ae --- /dev/null +++ b/griffin-doc/dev/dev-env-build.md @@ -0,0 +1,144 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + +# Apache Griffin Develop Environment Build Guide +For Griffin developers, you need to build an entire environment for development, it wastes time. +You can directly use the Griffin docker image, to the functional test during your development. +Please follow this [griffin docker guide](../docker/griffin-docker-guide.md). + +## Preparation +Follow this [griffin docker guide](../docker/griffin-docker-guide.md#environment-preparation). + +## Start in batch mode or streaming mode +Usually, you can run the docker image in batch mode(always the first choice) or streaming mode(only if you need a streaming environment like kafka data source), choose a mode to start. +For [Batch mode](../docker/griffin-docker-guide.md#how-to-use-griffin-docker-images-in-batch-mode) and [Streaming mode](../docker/griffin-docker-guide.md#how-to-use-griffin-docker-images-in-streaming-mode), follow step 1 and 2, you can start the docker container. + +## Run or Debug at local +### For service module +If you need to develop the service module, you need to modify some configuration in the following files. +Docker host is your machine running the docker containers, which means if you install docker and run docker containers on 1.2.3.4, the <docker host ip> is 1.2.3.4. + +In service/src/main/resources/application.properties +``` +spring.datasource.url = jdbc:postgresql://<docker host ip>:35432/quartz?autoReconnect=true&useSSL=false + +hive.metastore.uris = thrift://<docker host ip>:39083 + +elasticsearch.host = <docker host ip> +elasticsearch.port = 39200 +``` + +In service/src/main/resources/sparkJob.properties +``` +livy.uri=http://<docker host ip>:38998/batches + +spark.uri=http://<docker host ip>:38088 +``` + +Now you can start the service module in your local IDE, by running or debugging org.apache.griffin.core.GriffinWebApplication. + +### For ui module +If you need to develop the ui module only, you need to modify some configuration. + +In ui/angular/src/app/service/service.service.ts +``` +// public BACKEND_SERVER = ""; +public BACKEND_SERVER = 'http://<docker host ip>:38080'; +``` +After this, you can test your ui module by using remote service. + +However, in most conditions, you need to develop the ui module with some modification in service module. +Then you need to follow the steps above for service module first, and +In ui/angular/src/app/service/service.service.ts +``` +// public BACKEND_SERVER = ""; +public BACKEND_SERVER = 'http://localhost:8080'; +``` +After this, you can start service module at local, and test your ui module by using local service. + +### For measure module +If you need to develop the measure module only, you can ignore any of the service or ui module. +You can test your built measure JAR in the docker container, using the existed spark environment. + +For debug phase, you'd better install hadoop, spark, hive at local, and test your program at local for fast. + +## Deploy on docker container +First, in the incubator-griffin directory, build you packages at once. +``` +mvn clean install +``` + +### For service module and ui module +1. Login to docker container, and stop running griffin service. +``` +docker exec -it <griffin docker container id> bash +cd ~/service +ps -ef | grep service.jar +kill -9 <pid of service.jar> +``` +2. Service and ui module are both packaged in service/target/service-<version>.jar, copy it into your docker container. +``` +docker cp service-<version>.jar <griffin docker container id>:/root/service/service.jar +``` +3. In docker container, start the new service. +``` +cd ~/service +nohup java -jar service.jar > service.log & +``` +Now you can follow the service log by `tail -f service.log`. + +### For measure module +1. Measure module is packaged in measure/target/measure-<version>.jar, copy it into your docker container. +``` +docker cp measure-<version>.jar <griffin docker container id>:/root/measure/griffin-measure.jar +``` +2. Login to docker container, and overwrite griffin-measure.jar onto hdfs inside. +``` +docker exec -it <griffin docker container id> bash +hadoop fs -rm /griffin/griffin-measure.jar +hadoop fs -put /root/measure/griffin-measure.jar /griffin/griffin-measure.jar +``` +Now the griffin service will submit jobs by using this new griffin-measure.jar. + +## Build new griffin docker image +For end2end test, you will need to build a new griffin docker image, for more convenient test. +1. Pull the docker build repo on your docker host. +``` +git clone https://github.com/bhlx3lyx7/griffin-docker.git +``` +2. Copy your measure and service JAR into svc_msr_new directory. +``` +cp service-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/service/service.jar +cp measure-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/measure/griffin-measure.jar +``` +3. Build your new griffin docker image. +In svc_msr_new directory. +``` +cd <path to>/griffin-docker/svc_msr_new +docker build -t <image name>[:<image version>] . +``` +4. If you are using another image name (or version), you need also modify the docker-compose file you're using. +``` +griffin: + image: <image name>[:<image version>] +``` +5. Now you can run your new griffin docker image. +``` +docker-compose -f <docker-compose file> up -d +``` \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/293217f7/griffin-doc/docker/svc_msr/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-batch.yml b/griffin-doc/docker/svc_msr/docker-compose-batch.yml index 6c1cd49..5375b5c 100644 --- a/griffin-doc/docker/svc_msr/docker-compose-batch.yml +++ b/griffin-doc/docker/svc_msr/docker-compose-batch.yml @@ -28,6 +28,7 @@ griffin: - 32122:2122 - 38088:8088 - 33306:3306 + - 35432:5432 - 38042:8042 - 39083:9083 - 38998:8998 http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/293217f7/griffin-doc/docker/svc_msr/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml index 3c5280f..bb17f70 100644 --- a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml +++ b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml @@ -32,6 +32,7 @@ griffin: - 32122:2122 - 38088:8088 - 33306:3306 + - 35432:5432 - 38042:8042 - 39083:9083 - 38998:8998
