document modified to align with spark 2 and hive 2
Project: http://git-wip-us.apache.org/repos/asf/incubator-griffin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-griffin/commit/0ba7e074 Tree: http://git-wip-us.apache.org/repos/asf/incubator-griffin/tree/0ba7e074 Diff: http://git-wip-us.apache.org/repos/asf/incubator-griffin/diff/0ba7e074 Branch: refs/heads/griffin-0.2.0-incubating-rc4 Commit: 0ba7e07410f1a45c545c08ac998ef18c7840d16e Parents: ccb9110 Author: Lionel Liu <[email protected]> Authored: Tue Apr 24 17:08:09 2018 +0800 Committer: Lionel Liu <[email protected]> Committed: Tue Apr 24 17:08:09 2018 +0800 ---------------------------------------------------------------------- griffin-doc/deploy/deploy-guide.md | 42 ++++++++----- griffin-doc/dev/dev-env-build.md | 10 +-- .../docker/compose/docker-compose-batch.yml | 45 ++++++++++++++ .../docker/compose/docker-compose-streaming.yml | 65 ++++++++++++++++++++ griffin-doc/docker/griffin-docker-guide.md | 14 ++--- .../docker/svc_msr/docker-compose-batch.yml | 45 -------------- .../docker/svc_msr/docker-compose-streaming.yml | 65 -------------------- griffin-doc/roadmap.md | 41 +++++------- .../service/hibernate_eclipselink_switch.md | 19 ++++++ griffin-doc/service/mysql_postgresql_switch.md | 19 ++++++ service/src/main/resources/sparkJob.properties | 2 +- 11 files changed, 203 insertions(+), 164 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/deploy/deploy-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/deploy/deploy-guide.md b/griffin-doc/deploy/deploy-guide.md index 0693c25..d23ee15 100644 --- a/griffin-doc/deploy/deploy-guide.md +++ b/griffin-doc/deploy/deploy-guide.md @@ -23,11 +23,11 @@ For Griffin users, you can deploy it with some dependencies in your environment, ### Prerequisites You need to install following items - jdk (1.8 or later versions). -- mysql or Postgresql. +- Postgresql or Mysql. - npm (version 6.0.0+). - [Hadoop](http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz) (2.6.0 or later), you can get some help [here](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html). -- [Spark](http://spark.apache.org/downloads.html) (version 1.6.x, griffin does not support 2.0.x at current), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help [here](http://why-not-learn-something.blogspot.com/2015/06/spark-installation-pseudo.html). -- [Hive](http://apache.claz.org/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz) (version 1.2.1 or later), you can get some help [here](https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-RunningHive). +- [Spark](http://spark.apache.org/downloads.html) (version 2.2.1), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help [here](http://why-not-learn-something.blogspot.com/2015/06/spark-installation-pseudo.html). +- [Hive](http://apache.claz.org/hive/hive-2.2.0/apache-hive-2.2.0-bin.tar.gz) (version 2.2.0), you can get some help [here](https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-RunningHive). You need to make sure that your spark cluster could access your HiveContext. - [Livy](http://archive.cloudera.com/beta/livy/livy-server-0.3.0.zip), you can get some help [here](http://livy.io/quickstart.html). Griffin need to schedule spark jobs by server, we use livy to submit our jobs. @@ -37,18 +37,31 @@ You need to install following items datanucleus-core-3.2.10.jar datanucleus-rdbms-3.2.9.jar ``` -- ElasticSearch. +- ElasticSearch (5.0 or later). ElasticSearch works as a metrics collector, Griffin produces metrics to it, and our default UI get metrics from it, you can use your own way as well. ### Configuration +#### Postgresql + +Create database 'quartz' in postgresql +``` +createdb -O <username> quartz +``` +Init quartz tables in postgresql by [init_quartz.sql](../../service/src/main/resources/Init_quartz_postgres.sql) +``` +psql -p <password> -h <host address> -U <username> -f init_quartz.sql quartz +``` + +#### Mysql + Create database 'quartz' in mysql ``` -mysql -u username -e "create database quartz" -p +mysql -u <username> -e "create database quartz" -p ``` -Init quartz tables in mysql by service/src/main/resources/Init_quartz.sql +Init quartz tables in mysql by [init_quartz.sql](../../service/src/main/resources/Init_quartz_mysql.sql) ``` -mysql -u username -p quartz < service/src/main/resources/Init_quartz.sql +mysql -u <username> -p quartz < init_quartz.sql ``` @@ -112,20 +125,17 @@ You should also modify some configurations of Griffin for your environment. sparkJob.file = hdfs://<griffin measure path>/griffin-measure.jar sparkJob.args_1 = hdfs://<griffin env path>/env.json - sparkJob.jars = hdfs://<datanucleus path>/spark-avro_2.11-2.0.1.jar\ - hdfs://<datanucleus path>/datanucleus-api-jdo-3.2.6.jar\ - hdfs://<datanucleus path>/datanucleus-core-3.2.10.jar\ - hdfs://<datanucleus path>/datanucleus-rdbms-3.2.9.jar + # other dependent jars + sparkJob.jars = - spark.yarn.dist.files = hdfs:///<spark conf path>/hive-site.xml + # hive-site.xml location, as configured in spark conf if ignored here + spark.yarn.dist.files = livy.uri = http://<your IP>:8998/batches spark.uri = http://<your IP>:8088 ``` - \<griffin measure path> is the location you should put the jar file of measure module. - \<griffin env path> is the location you should put the env.json file. - - \<datanucleus path> is the location you should put the 3 jar files of livy, and the spark avro jar file if you need to support avro data. - - \<spark conf path> is the location of spark conf directory. ### Build and Run @@ -154,7 +164,7 @@ After a few seconds, we can visit our default UI of Griffin (by default the port http://<your IP>:8080 ``` -You can use UI following the steps [here](../ui/user-guide.md). +You can use UI following the steps [here](../ui/user-guide.md). -**Note**: The front-end UI is still under development, you can only access some basic features currently. +**Note**: The UI doesn't support all the features, for the advanced features you can try API of service. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/dev/dev-env-build.md ---------------------------------------------------------------------- diff --git a/griffin-doc/dev/dev-env-build.md b/griffin-doc/dev/dev-env-build.md index fc981cb..4e075f8 100644 --- a/griffin-doc/dev/dev-env-build.md +++ b/griffin-doc/dev/dev-env-build.md @@ -122,15 +122,15 @@ For end2end test, you will need to build a new griffin docker image, for more co ``` git clone https://github.com/bhlx3lyx7/griffin-docker.git ``` -2. Copy your measure and service JAR into svc_msr_new directory. +2. Copy your measure and service JAR into griffin_spark2 directory. ``` -cp service-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/service/service.jar -cp measure-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/measure/griffin-measure.jar +cp service-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/service/service.jar +cp measure-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/measure/griffin-measure.jar ``` 3. Build your new griffin docker image. -In svc_msr_new directory. +In griffin_spark2 directory. ``` -cd <path to>/griffin-docker/svc_msr_new +cd <path to>/griffin-docker/griffin_spark2 docker build -t <image name>[:<image version>] . ``` 4. If you are using another image name (or version), you need also modify the docker-compose file you're using. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/docker/compose/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/compose/docker-compose-batch.yml b/griffin-doc/docker/compose/docker-compose-batch.yml new file mode 100644 index 0000000..9b247f2 --- /dev/null +++ b/griffin-doc/docker/compose/docker-compose-batch.yml @@ -0,0 +1,45 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/griffin_spark2:0.2.0 + hostname: griffin + links: + - es + environment: + ES_HOSTNAME: es + volumes: + - /var/lib/mysql + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 35432:5432 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + - 39300:9300 + container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/docker/compose/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/compose/docker-compose-streaming.yml b/griffin-doc/docker/compose/docker-compose-streaming.yml new file mode 100644 index 0000000..44759e0 --- /dev/null +++ b/griffin-doc/docker/compose/docker-compose-streaming.yml @@ -0,0 +1,65 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/griffin_spark2:0.2.0 + hostname: griffin + links: + - es + - zk + - kafka + environment: + ES_HOSTNAME: es + ZK_HOSTNAME: zk + KAFKA_HOSTNAME: kafka + volumes: + - /var/lib/mysql + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 35432:5432 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + - 39300:9300 + container_name: es + +zk: + image: zookeeper:3.5 + hostname: zk + ports: + - 32181:2181 + container_name: zk + restart: always + +kafka: + image: bhlx3lyx7/kafka + hostname: kafka + ports: + - 39092:9092 + container_name: kafka + tty: true \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/docker/griffin-docker-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/griffin-docker-guide.md b/griffin-doc/docker/griffin-docker-guide.md index bc36759..9308e7b 100644 --- a/griffin-doc/docker/griffin-docker-guide.md +++ b/griffin-doc/docker/griffin-docker-guide.md @@ -30,32 +30,32 @@ Griffin docker images are pre-built on docker hub, users can pull them to try gr ``` 3. Pull griffin pre-built docker images. ``` - docker pull bhlx3lyx7/svc_msr:0.2.0 + docker pull bhlx3lyx7/griffin_spark2:0.2.0 docker pull bhlx3lyx7/elasticsearch docker pull bhlx3lyx7/kafka docker pull zookeeper:3.5 ``` Or you can pull the images faster through mirror acceleration if you are in China. ``` - docker pull registry.docker-cn.com/bhlx3lyx7/svc_msr:0.2.0 + docker pull registry.docker-cn.com/bhlx3lyx7/griffin_spark2:0.2.0 docker pull registry.docker-cn.com/bhlx3lyx7/elasticsearch docker pull registry.docker-cn.com/bhlx3lyx7/kafka docker pull registry.docker-cn.com/zookeeper:3.5 ``` The docker images are the griffin environment images. - - `bhlx3lyx7/svc_msr`: This image contains mysql, hadoop, hive, spark, livy, griffin service, griffin measure, and some prepared demo data, it works as a single node spark cluster, providing spark engine and griffin service. + - `bhlx3lyx7/griffin_spark2`: This image contains mysql, hadoop, hive, spark, livy, griffin service, griffin measure, and some prepared demo data, it works as a single node spark cluster, providing spark engine and griffin service. - `bhlx3lyx7/elasticsearch`: This image is based on official elasticsearch, adding some configurations to enable cors requests, to provide elasticsearch service for metrics persist. - `bhlx3lyx7/kafka`: This image contains kafka 0.8, and some demo streaming data, to provide streaming data source in streaming mode. - `zookeeper:3.5`: This image is official zookeeper, to provide zookeeper service in streaming mode. ### How to use griffin docker images in batch mode -1. Copy [docker-compose-batch.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-batch.yml) to your work path. +1. Copy [docker-compose-batch.yml](compose/docker-compose-batch.yml) to your work path. 2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. ``` docker-compose -f docker-compose-batch.yml up -d ``` -3. Now you can try griffin APIs by using postman after importing the [json files](https://github.com/apache/incubator-griffin/tree/master/griffin-doc/service/postman). - In which you need to modify the environment `BASE_PATH` value into `<your local IP address>:38080`. +3. Now you can try griffin APIs by using postman after importing the [json files](../service/postman). + In which you need to modify the environment `BASE_PATH` value to `<your local IP address>:38080`. 4. You can try the api `Basic -> Get griffin version`, to make sure griffin service has started up. 5. Add an accuracy measure through api `Measures -> Add measure`, to create a measure in griffin. 6. Add a job to through api `jobs -> Add job`, to schedule a job to execute the measure. In the example, the schedule interval is 5 minutes. @@ -65,7 +65,7 @@ Griffin docker images are pre-built on docker hub, users can pull them to try gr ``` ### How to use griffin docker images in streaming mode -1. Copy [docker-compose-streaming.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-streaming.yml) to your work path. +1. Copy [docker-compose-streaming.yml](compose/docker-compose-streaming.yml) to your work path. 2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. ``` docker-compose -f docker-compose-streaming.yml up -d http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/docker/svc_msr/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-batch.yml b/griffin-doc/docker/svc_msr/docker-compose-batch.yml deleted file mode 100644 index 5375b5c..0000000 --- a/griffin-doc/docker/svc_msr/docker-compose-batch.yml +++ /dev/null @@ -1,45 +0,0 @@ -#Licensed to the Apache Software Foundation (ASF) under one -#or more contributor license agreements. See the NOTICE file -#distributed with this work for additional information -#regarding copyright ownership. The ASF licenses this file -#to you under the Apache License, Version 2.0 (the -#"License"); you may not use this file except in compliance -#with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -#Unless required by applicable law or agreed to in writing, -#software distributed under the License is distributed on an -#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -#KIND, either express or implied. See the License for the -#specific language governing permissions and limitations -#under the License. - -griffin: - image: bhlx3lyx7/svc_msr:0.2.0 - hostname: griffin - links: - - es - environment: - ES_HOSTNAME: es - volumes: - - /var/lib/mysql - ports: - - 32122:2122 - - 38088:8088 - - 33306:3306 - - 35432:5432 - - 38042:8042 - - 39083:9083 - - 38998:8998 - - 38080:8080 - tty: true - container_name: griffin - -es: - image: bhlx3lyx7/elasticsearch - hostname: es - ports: - - 39200:9200 - - 39300:9300 - container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/docker/svc_msr/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml deleted file mode 100644 index bb17f70..0000000 --- a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml +++ /dev/null @@ -1,65 +0,0 @@ -#Licensed to the Apache Software Foundation (ASF) under one -#or more contributor license agreements. See the NOTICE file -#distributed with this work for additional information -#regarding copyright ownership. The ASF licenses this file -#to you under the Apache License, Version 2.0 (the -#"License"); you may not use this file except in compliance -#with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -#Unless required by applicable law or agreed to in writing, -#software distributed under the License is distributed on an -#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -#KIND, either express or implied. See the License for the -#specific language governing permissions and limitations -#under the License. - -griffin: - image: bhlx3lyx7/svc_msr:0.2.0 - hostname: griffin - links: - - es - - zk - - kafka - environment: - ES_HOSTNAME: es - ZK_HOSTNAME: zk - KAFKA_HOSTNAME: kafka - volumes: - - /var/lib/mysql - ports: - - 32122:2122 - - 38088:8088 - - 33306:3306 - - 35432:5432 - - 38042:8042 - - 39083:9083 - - 38998:8998 - - 38080:8080 - tty: true - container_name: griffin - -es: - image: bhlx3lyx7/elasticsearch - hostname: es - ports: - - 39200:9200 - - 39300:9300 - container_name: es - -zk: - image: zookeeper:3.5 - hostname: zk - ports: - - 32181:2181 - container_name: zk - restart: always - -kafka: - image: bhlx3lyx7/kafka - hostname: kafka - ports: - - 39092:9092 - container_name: kafka - tty: true \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/roadmap.md ---------------------------------------------------------------------- diff --git a/griffin-doc/roadmap.md b/griffin-doc/roadmap.md index b916311..01807e8 100644 --- a/griffin-doc/roadmap.md +++ b/griffin-doc/roadmap.md @@ -21,41 +21,32 @@ under the License. ## Current feature list In the current version, we've implemented the below main DQ features -- **Data Asset Management** - - User can register, delete, edit data assets, currently only Hadoop data-sets are supported +- **Data Asset Detection** + After configuration in service module, Griffin can detect the Hive tables metadata through Hive metastore service. - **Measure Management** + Through UI, user can create, delete measures for 3 types: accuracy, profiling and publish metrics. + Through service API, user can create, delete and update measures for 6 types: accuracy, profiling, timeliness, uniqueness, completeness and publish metrics. - User can create, delete, edit measures for 4 types: Accuracy, Profiling, Anomaly Detection, Publish Metrics - -- **Job Scheduler** - - After the measures are created, the Job Scheduler component can create the jobs and schedule them to calculate the metrics values +- **Job Management** + User can create, delete job to schedule batch job for calculative measures, data range of each calculation, and the extra trigger condition like "done file" on hdfs. -- **Measure Execution on Spark** - - The Job Scheduler will trigger the measure execution on Spark to generate the metrics values +- **Measure Calculation on Spark** + Service module will trigger and submit calculation jobs to Spark cluster through livy, the measure module calculates and persists the metric values to elasticsearch by default. - **Metrics Visualization** - - We have a web portal to display all metrics - -- **My Dashboard** - - Only the interested metrics will be displayed on "My Dashboard" + Through service API, user can get metric values of each job from elasticsearch. + On UI, accuracy metrics will be rendered as a chart, profiling metrics will be displayed as a table. ## Short-term Roadmap -- **Support more data-set types** - - Current we only support Hadoop datasets, we should also support RDBMS and real-time streaming data from Kafka, Storm, etc. +- **Support more data source types** + At current, Griffin only supports Hive table, avro files on hdfs as data source in batch mode, Kafka as data source in streaming mode. + We plan to support more data source types, like RDBM, elasticsearch. - **Support more data quality dimensions** + Griffin need to support more data quality dimensions, like consistency and validity. - Besides accuracy, there are some other data quality dimensions(Completeness, Uniqueness, Timeliness, Validity, Consistency), we should support more dimensions - -- **More ML algorithms for Anomaly Detection measure** - - Currently only [MAD(Median absolute deviation)](https://en.wikipedia.org/wiki/Median_absolute_deviation) and [Bollinger Bands](https://en.wikipedia.org/wiki/Bollinger_Bands) are supported, we are considering to support more Machine Learning algorithms +- **Anomaly Detection** + Griffin plan to support anomaly detection, by analyzing calculated metrics from elasticsearch. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/service/hibernate_eclipselink_switch.md ---------------------------------------------------------------------- diff --git a/griffin-doc/service/hibernate_eclipselink_switch.md b/griffin-doc/service/hibernate_eclipselink_switch.md index 3c1ad06..6bfae3b 100644 --- a/griffin-doc/service/hibernate_eclipselink_switch.md +++ b/griffin-doc/service/hibernate_eclipselink_switch.md @@ -1,3 +1,22 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + # Hibernate and eclipselink switch ## Overview In this document,we list two main part. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/griffin-doc/service/mysql_postgresql_switch.md ---------------------------------------------------------------------- diff --git a/griffin-doc/service/mysql_postgresql_switch.md b/griffin-doc/service/mysql_postgresql_switch.md index d3925a1..081307c 100644 --- a/griffin-doc/service/mysql_postgresql_switch.md +++ b/griffin-doc/service/mysql_postgresql_switch.md @@ -1,3 +1,22 @@ +<!-- +Licensed to the Apache Software Foundation (ASF) under one +or more contributor license agreements. See the NOTICE file +distributed with this work for additional information +regarding copyright ownership. The ASF licenses this file +to you under the Apache License, Version 2.0 (the +"License"); you may not use this file except in compliance +with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, +software distributed under the License is distributed on an +"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +KIND, either express or implied. See the License for the +specific language governing permissions and limitations +under the License. +--> + # Mysql and postgresql switch ## Overview http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/0ba7e074/service/src/main/resources/sparkJob.properties ---------------------------------------------------------------------- diff --git a/service/src/main/resources/sparkJob.properties b/service/src/main/resources/sparkJob.properties index ce2587a..1334747 100644 --- a/service/src/main/resources/sparkJob.properties +++ b/service/src/main/resources/sparkJob.properties @@ -32,7 +32,7 @@ sparkJob.executorCores=1 sparkJob.driverMemory=1g sparkJob.executorMemory=1g -# shouldn't config in server, but in +# other dependent jars sparkJob.jars = # hive-site.xml location, as configured in spark conf if ignored here
