[GRIFFIN-163] merge version 0.2.0 for spark 2 into master branch Author: Lionel Liu <[email protected]> Author: dodobel <[email protected]>
Closes #282 from bhlx3lyx7/master-merge. Project: http://git-wip-us.apache.org/repos/asf/incubator-griffin/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-griffin/commit/47ae1774 Tree: http://git-wip-us.apache.org/repos/asf/incubator-griffin/tree/47ae1774 Diff: http://git-wip-us.apache.org/repos/asf/incubator-griffin/diff/47ae1774 Branch: refs/heads/master Commit: 47ae177487c9596845cc9efcde173857c5ce7a19 Parents: 359bfe2 Author: Lionel Liu <[email protected]> Authored: Thu May 17 10:53:32 2018 +0800 Committer: Lionel Liu <[email protected]> Committed: Thu May 17 10:53:32 2018 +0800 ---------------------------------------------------------------------- LICENSE | 265 ++---------- griffin-doc/deploy/deploy-guide.md | 42 +- griffin-doc/dev/dev-env-build.md | 10 +- .../docker/compose/docker-compose-batch.yml | 45 ++ .../docker/compose/docker-compose-streaming.yml | 65 +++ griffin-doc/docker/griffin-docker-guide.md | 42 +- .../docker/svc_msr/docker-compose-batch.yml | 45 -- .../docker/svc_msr/docker-compose-streaming.yml | 65 --- griffin-doc/measure/dsl-guide.md | 26 +- .../measure/measure-configuration-guide.md | 4 +- griffin-doc/roadmap.md | 41 +- licenses/BSD-2-Clause | 22 - licenses/BSD-3-Clause | 26 -- licenses/CDDL-1.0 | 363 ---------------- licenses/CDDL-1.1 | 362 ---------------- licenses/EPL | 86 ---- licenses/JSON | 26 -- licenses/MIT | 19 - licenses/NOTICE-Jackson | 20 - licenses/NOTICE-Tomcat | 55 --- licenses/PostgreSQL | 12 - licenses/SIL-OFL-1.1 | 84 ---- licenses/WTFPL | 13 - measure/pom.xml | 16 +- measure/src/main/resources/META-INF/LICENSE | 256 ------------ measure/src/main/resources/META-INF/NOTICE | 10 - .../batch/HiveBatchDataConnector.scala | 6 +- .../data/source/cache/DataSourceCache.scala | 2 +- .../data/source/cache/JsonDataSourceCache.scala | 4 +- .../data/source/cache/OrcDataSourceCache.scala | 4 +- .../source/cache/ParquetDataSourceCache.scala | 4 +- .../griffin/measure/persist/MultiPersists.scala | 2 +- .../griffin/measure/persist/Persist.scala | 2 +- .../measure/process/BatchDqProcess.scala | 23 +- .../measure/process/StreamingDqProcess.scala | 22 +- .../process/engine/DataFrameOprEngine.scala | 10 +- .../measure/process/engine/DqEngine.scala | 2 +- .../measure/process/engine/DqEngines.scala | 2 +- .../measure/process/engine/SparkDqEngine.scala | 9 +- .../measure/process/engine/SparkSqlEngine.scala | 2 +- .../measure/rule/adaptor/RuleAdaptorGroup.scala | 17 +- .../measure/rule/dsl/expr/SelectExpr.scala | 2 +- .../griffin/measure/util/MessageUtil.java | 320 +++++++------- .../_accuracy-streaming-griffindsl.json | 8 +- .../_completeness-streaming-griffindsl.json | 7 +- .../_profiling-batch-griffindsl-hive.json | 48 +++ .../_profiling-streaming-griffindsl.json | 7 +- measure/src/test/resources/env-streaming.json | 1 + service/pom.xml | 35 +- .../apache/griffin/core/job/SparkSubmitJob.java | 7 +- .../core/metastore/hive/HiveMetaStoreProxy.java | 6 +- .../hive/HiveMetaStoreServiceImpl.java | 6 +- service/src/main/resources/META-INF/LICENSE | 414 ------------------- service/src/main/resources/META-INF/NOTICE | 15 - service/src/main/resources/hive-site.xml | 22 + .../src/main/resources/hivemetastore-site.xml | 22 + service/src/main/resources/hiveserver2-site.xml | 22 + service/src/main/resources/sparkJob.properties | 10 +- ui/LICENSE | 310 -------------- ui/NOTICE | 5 - ui/angular/src/app/service/service.service.ts | 2 +- 61 files changed, 631 insertions(+), 2769 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/LICENSE ---------------------------------------------------------------------- diff --git a/LICENSE b/LICENSE index eb94865..9c8f3ea 100644 --- a/LICENSE +++ b/LICENSE @@ -173,242 +173,29 @@ incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. ---------------------------------------------------- - -The Apache Griffin contains subcomponents in the source code -release with separate copyright notices and license terms. Your use of -the source code for the these subcomponents is subject to the terms and -conditions of their respective licenses. - ----------------------------------------------- -Public Domain ----------------------------------------------- -The following components are provided in Public Domain. See project link for details. - - (Public Domain) XZ for Java (org.tukaani:xz:1.0 - http://tukaani.org/xz/java.html) - (Public Domain) AOP alliance (aopalliance:aopalliance:1.0 - http://aopalliance.sourceforge.net) - (Public Domain) XZ for Java (org.tukaani:xz:1.0 - http://tukaani.org/xz/java.html) - ----------------------------------------------- -JSON license ----------------------------------------------- -The following components are provided under a JSON license. See project link for details. -The text of each license is also included at licenses/JSON. - - (The JSON License) JSON in Java (org.json:json:20140107 - https://github.com/douglascrockford/JSON-java) - ----------------------------------------------- -The PostgreSQL License ----------------------------------------------- -The following components are provided under a PostgreSQL license. See project link for details. -The text of each license is also included at licenses/PostgreSQL. - - (The PostgreSQL License) PostgreSQL JDBC Driver - JDBC 4.1 (org.postgresql:postgresql:9.4.1212.jre7 - https://github.com/pgjdbc/pgjdbc-parent-poms/pgjdbc-core-parent/pgjdbc-core-prevjre/postgresql) - ----------------------------------------------- -EPL license ----------------------------------------------- -The following components are provided under a EPL license. See project link for details. -The text of each license is also included at licenses/EPL. - - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) Javax Persistence (org.eclipse.persistence:javax.persistence:2.1.0 - http://www.eclipse.org/eclipselink) - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) EclipseLink ANTLR (org.eclipse.persistence:org.eclipse.persistence.antlr:2.6.0 - http://www.eclipse.org/eclipselink) - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) EclipseLink ASM (org.eclipse.persistence:org.eclipse.persistence.asm:2.6.0 - http://www.eclipse.org/eclipselink) - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) EclipseLink Core (org.eclipse.persistence:org.eclipse.persistence.core:2.6.0 - http://www.eclipse.org/eclipselink) - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) EclipseLink JPA (org.eclipse.persistence:org.eclipse.persistence.jpa:2.6.0 - http://www.eclipse.org/eclipselink) - (Eclipse Distribution License v. 1.0) (Eclipse Public License v1.0) EclipseLink Hermes JPQL Parser (org.eclipse.persistence:org.eclipse.persistence.jpa.jpql:2.6.0 - http://www.eclipse.org/eclipselink) - (Eclipse Public License - v 1.0) (GNU Lesser General Public License) Logback Classic Module (ch.qos.logback:logback-classic:1.1.9 - http://logback.qos.ch/logback-classic) - (Eclipse Public License - v 1.0) (GNU Lesser General Public License) Logback Core Module (ch.qos.logback:logback-core:1.1.9 - http://logback.qos.ch/logback-core) - (MPL 2.0 or EPL 1.0) H2 Database Engine (com.h2database:h2:1.4.193 - http://www.h2database.com) - (Eclipse Public License - v 1.0) AspectJ weaver (org.aspectj:aspectjweaver:1.8.9 - http://www.aspectj.org) - (GNU LESSER GENERAL PUBLIC LICENSE)(Eclipse Public License) c3p0:JDBC DataSources/Resource Pools (c3p0:c3p0:0.9.1.1 - http://c3p0.sourceforge.net) - ----------------------------------------------- -CDDL licenses ----------------------------------------------- -The following components are provided under a CDDL license. See project link for details. -The text of each license is also included at licenses/CDDL-1.0 and licenses/CDDL-1.1. - - (CDDL) (GPLv2+CE) JavaMail API (com.sun.mail:javax.mail:1.4.4 - http://kenai.com/projects/javamail/javax.mail) - (Common Development and Distribution License (CDDL) v1.0) JavaBeans Activation Framework (JAF) (javax.activation:activation:1.1 - http://java.sun.com/products/javabeans/jaf/index.jsp) - (CDDL license) jsp-api (javax.servlet.jsp:jsp-api:2.1 - no url defined) - (CDDL-1.0 license) Java Transaction API (javax.transaction:jta:1.1 - http://java.sun.com/products/jta) - (CDDL 1.1) (GPL2 w/ CPE) jersey-client (com.sun.jersey:jersey-client:1.9 - https://jersey.java.net/jersey-client/) - (CDDL 1.1) (GPL2 w/ CPE) jersey-core (com.sun.jersey:jersey-core:1.9 - https://jersey.java.net/jersey-core/) - (CDDL 1.1) (GPL2 w/ CPE) jersey-json (com.sun.jersey:jersey-json:1.9 - https://jersey.java.net/jersey-json/) - (CDDL 1.1) (GPL2 w/ CPE) jersey-server (com.sun.jersey:jersey-server:1.9 - https://jersey.java.net/jersey-server/) - (CDDL 1.1) (GPL2 w/ CPE) jersey-guice (com.sun.jersey.contribs:jersey-guice:1.9 - https://jersey.java.net/jersey-contribs/jersey-guice/) - (CDDL 1.1) (GPL2 w/ CPE) JAXB RI (com.sun.xml.bind:jaxb-impl:2.2.3-1 - http://jaxb.java.net/) - (Common Development and Distribution License (CDDL) v1.0) JavaBeans Activation Framework (JAF) (javax.activation:activation:1.1 - http://java.sun.com/products/javabeans/jaf/index.jsp) - (CDDL + GPLv2 with classpath exception) javax.transaction API (javax.transaction:javax.transaction-api:1.2 - http://jta-spec.java.net) - (CDDL 1.1) (GPL2 w/ CPE) JAXB API bundle for GlassFish V3 (javax.xml.bind:jaxb-api:2.2.2 - https://jaxb.dev.java.net/) - (COMMON DEVELOPMENT AND DISTRIBUTION LICENSE (CDDL) Version 1.0) (GNU General Public Library) Streaming API for XML (javax.xml.stream:stax-api:1.0-2 - no url defined) - ----------------------------------------------- -BSD-style licenses ----------------------------------------------- -The following components are provided under a BSD-style license. See project link for details. -The text of each license is also included at licenses/BSD-3-Clause and licenses/BSD-2-Clause. - - (BSD) ParaNamer Core (com.thoughtworks.paranamer:paranamer:2.3 - http://paranamer.codehaus.org/paranamer) - (BSD-like) Scala Library (org.scala-lang:scala-library:2.10.6 - http://www.scala-lang.org/) - (BSD-like) Scala Compiler (org.scala-lang:scala-reflect:2.10.6 - http://www.scala-lang.org/) - (BSD license) ASM Core (asm:asm:3.1 - http://asm.objectweb.org/asm/) - (BSD license) dom4j (dom4j:dom4j:1.6.1 - http://dom4j.org) - (BSD license) Antlr 3.4 Runtime (org.antlr:antlr-runtime:3.4 - http://www.antlr.org) - (BSD License) AntLR Parser Generator (antlr:antlr:2.7.7 - http://www.antlr.org/) - (New BSD License) Kryo (com.esotericsoftware.kryo:kryo:2.22 - http://code.google.com/p/kryo/) - (New BSD license) Protocol Buffer Java API (com.google.protobuf:protobuf-java:2.5.0 - http://code.google.com/p/protobuf) - (BSD) ParaNamer Core (com.thoughtworks.paranamer:paranamer:2.3 - http://paranamer.codehaus.org/paranamer) - (BSD licence) ANTLR StringTemplate (org.antlr:stringtemplate:3.2.1 - http://www.stringtemplate.org) - (The BSD 3-Clause License) leveldbjni-all (org.fusesource.leveldbjni:leveldbjni-all:1.8 - http://leveldbjni.fusesource.org/leveldbjni-all) - (BSD-like) Scala Library (org.scala-lang:scala-library:2.10.4 - http://www.scala-lang.org/) - (The BSD License) xmlenc Library (xmlenc:xmlenc:0.52 - http://xmlenc.sourceforge.net) - ----------------------------------------------- -MIT licenses ----------------------------------------------- -The following components are provided under a MIT license. See project link for details. -The text of each license is also included at licenses/MIT. - - (MIT License) SLF4J API Module (org.slf4j:slf4j-api:1.7.21 - http://www.slf4j.org) - (MIT License) SLF4J LOG4J-12 Binding (org.slf4j:slf4j-log4j12:1.7.21 - http://www.slf4j.org) - (MIT License) JCL 1.1.1 implemented over SLF4J (org.slf4j:jcl-over-slf4j:1.7.22 - http://www.slf4j.org) - (MIT License) JUL to SLF4J bridge (org.slf4j:jul-to-slf4j:1.7.22 - http://www.slf4j.org) - (MIT License) SLF4J API Module (org.slf4j:slf4j-api:1.7.22 - http://www.slf4j.org) - (MIT License) SLF4J LOG4J-12 Binding (org.slf4j:slf4j-log4j12:1.7.22 - http://www.slf4j.org) - - ---------------------------------------------------- - -This project includes the software: angular - Available at: https://angular.io - Version used: 4.4.4 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) 2014-2016 Google, Inc. - -This project includes the software: echarts - Available at: http://echarts.baidu.com - Version used: 3.7.0 - Used under the following license: The BSD 3-Clause (New BSD) License (http://opensource.org/licenses/BSD-3-Clause) - Copyright (c) 2013, Baidu Inc. - -This project includes the software: Font Awesome (code) - Available at: http://fontawesome.io - Inclusive of: font-awesome/{css,scss,less}/* - Version used: 4.7.0 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) Dave Gandy (2016) - -This project includes the software: Font Awesome (fonts) - Available at: http://fontawesome.io - Inclusive of: font-awesome/fonts/* - Version used: 4.7.0 - Used under the following license: SIL OFL 1.1 (http://scripts.sil.org/OFL) - See details in licenses/SIL-OFL-1.1 - Copyright (c) Dave Gandy (2016) - -This project includes the software: requirejs - Available at: http://requirejs.org - Version used: 2.2.0 - Used under the following license: The BSD 3-Clause (New BSD) License (http://opensource.org/licenses/BSD-3-Clause) - Copyright (c) 2010-2015, The Dojo Foundation - -This project includes the software: jQuery JavaScript Library - Available at: http://jquery.com - Developed by: The jQuery Foundation (http://jquery.org/) - Inclusive of: jquery.js - Version used: 3.2.1 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright 2018 The jQuery Foundation - -This project includes the software: angular2-multiselect-dropdown - Available at: http://cuppalabs.github.io/components/multiselectDropdown - Version used: 1.3.4 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) 2009-2017, Cuppa Labs. All rights reserved and copyrighted to Pradeep Kumar Terli. - -This project includes the software: angular2-toaster - Available at: https://github.com/Stabzs/Angular2-Toaster - Version used: 4.0.1 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) 2016-2017 Stabzs. - -This project includes the software: Twitter Bootstrap - Available at: https://getbootstrap.com - Version used: 3.3.7 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) 2011-2014 Twitter, Inc - -This project includes the software: Glyphicons Halflings(glyphicons-halflings-regular.*) - Available at: http://glyphicons.com - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - Copyright (c) 2010 - 2016 Jan Kovarik - -This project includes the software: nouislider - Available at: https://refreshless.com/nouislider - Version used: 11.0.3 - Used under the following license: WTFPL - See details in licenses/WTFPL - -This project includes the software: ng2-nouislider - Available at: http://tb.github.io/ng2-nouislider - Version used: 1.7.6 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: angular-tree-component - Available at: https://angular2-tree.readme.io/docs - Version used: 4.1.0 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: angular2-datatable - Available at: https://github.com/mariuszfoltak/angular2-datatable - Version used: 0.6.0 - Used under the following license: Apache License, version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) - Copyright (c) Twitter, Inc. (2012) - -This project includes the software: rxjs - Available at: http://reactivex.io/rxjs - Version used: 5.4.2 - Used under the following license: Apache License, version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) - Copyright (c) Twitter, Inc. (2012) - -This project includes the software: webpack - Available at: https://webpack.js.org - Version used: 3.5.4 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: css-loader - Available at: https://github.com/webpack-contrib/css-loader - Version used: 0.28.7 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: lodash - Available at: https://github.com/lodash/lodash - Version used: 4.17.4 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: mobx-angular - Available at: https://github.com/mobxjs/mobx-angular - Version used: 1.5.0 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: mobx - Available at: https://mobx.js.org/ - Version used: 3.1.11 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: style-loader - Available at: https://github.com/webpack-contrib/style-loader - Version used: 0.13.2 - Used under the following license: The MIT License (http://opensource.org/licenses/MIT) - -This project includes the software: tslib - Available at: http://www.tslib.org/ - Version used: 0.13.2 - Used under the following license: Apache License, version 2.0 (http://www.apache.org/licenses/LICENSE-2.0) - Copyright (c) Twitter, Inc. (2012) - ---------------------------------------------------- - -All the dependent components licenses are listed in licenses/ directory. \ No newline at end of file + END OF TERMS AND CONDITIONS + + APPENDIX: How to apply the Apache License to your work. + + To apply the Apache License to your work, attach the following + boilerplate notice, with the fields enclosed by brackets "{}" + replaced with your own identifying information. (Don't include + the brackets!) The text should be enclosed in the appropriate + comment syntax for the file format. We also recommend that a + file or class name and description of purpose be included on the + same "printed page" as the copyright notice for easier + identification within third-party archives. + + Copyright {yyyy} {name of copyright owner} + + Licensed under the Apache License, Version 2.0 (the "License"); + you may not use this file except in compliance with the License. + You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/deploy/deploy-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/deploy/deploy-guide.md b/griffin-doc/deploy/deploy-guide.md index 7894e80..5e112a3 100644 --- a/griffin-doc/deploy/deploy-guide.md +++ b/griffin-doc/deploy/deploy-guide.md @@ -23,11 +23,11 @@ For Griffin users, please follow the instructions below to deploy Griffin in you ### Prerequisites You need to install following items - jdk (1.8 or later versions). -- mysql or Postgresql. +- Postgresql or Mysql. - npm (version 6.0.0+). - [Hadoop](http://apache.claz.org/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz) (2.6.0 or later), you can get some help [here](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/SingleCluster.html). -- [Spark](http://spark.apache.org/downloads.html) (version 1.6.x, griffin does not support 2.0.x at current), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help [here](http://why-not-learn-something.blogspot.com/2015/06/spark-installation-pseudo.html). -- [Hive](http://apache.claz.org/hive/hive-1.2.1/apache-hive-1.2.1-bin.tar.gz) (version 1.2.1 or later), you can get some help [here](https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-RunningHive). +- [Spark](http://spark.apache.org/downloads.html) (version 2.2.1), if you want to install Pseudo Distributed/Single Node Cluster, you can get some help [here](http://why-not-learn-something.blogspot.com/2015/06/spark-installation-pseudo.html). +- [Hive](http://apache.claz.org/hive/hive-2.2.0/apache-hive-2.2.0-bin.tar.gz) (version 2.2.0), you can get some help [here](https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-RunningHive). You need to make sure that your spark cluster could access your HiveContext. - [Livy](http://archive.cloudera.com/beta/livy/livy-server-0.3.0.zip), you can get some help [here](http://livy.io/quickstart.html). Griffin need to schedule spark jobs by server, we use livy to submit our jobs. @@ -37,18 +37,31 @@ You need to install following items datanucleus-core-3.2.10.jar datanucleus-rdbms-3.2.9.jar ``` -- ElasticSearch. +- ElasticSearch (5.0 or later). ElasticSearch works as a metrics collector, Griffin produces metrics to it, and our default UI get metrics from it, you can use your own way as well. ### Configuration +#### Postgresql + +Create database 'quartz' in postgresql +``` +createdb -O <username> quartz +``` +Init quartz tables in postgresql by [init_quartz.sql](../../service/src/main/resources/Init_quartz_postgres.sql) +``` +psql -p <password> -h <host address> -U <username> -f init_quartz.sql quartz +``` + +#### Mysql + Create database 'quartz' in mysql ``` -mysql -u username -e "create database quartz" -p +mysql -u <username> -e "create database quartz" -p ``` -Init quartz tables in mysql by service/src/main/resources/Init_quartz.sql +Init quartz tables in mysql by [init_quartz.sql](../../service/src/main/resources/Init_quartz_mysql.sql) ``` -mysql -u username -p quartz < service/src/main/resources/Init_quartz.sql +mysql -u <username> -p quartz < init_quartz.sql ``` @@ -112,20 +125,17 @@ You should also modify some configurations of Griffin for your environment. sparkJob.file = hdfs://<griffin measure path>/griffin-measure.jar sparkJob.args_1 = hdfs://<griffin env path>/env.json - sparkJob.jars = hdfs://<datanucleus path>/spark-avro_2.11-2.0.1.jar\ - hdfs://<datanucleus path>/datanucleus-api-jdo-3.2.6.jar\ - hdfs://<datanucleus path>/datanucleus-core-3.2.10.jar\ - hdfs://<datanucleus path>/datanucleus-rdbms-3.2.9.jar + # other dependent jars + sparkJob.jars = - spark.yarn.dist.files = hdfs:///<spark conf path>/hive-site.xml + # hive-site.xml location + spark.yarn.dist.files = hdfs://<path to>/hive-site.xml livy.uri = http://<your IP>:8998/batches spark.uri = http://<your IP>:8088 ``` - \<griffin measure path> is the location you should put the jar file of measure module. - \<griffin env path> is the location you should put the env.json file. - - \<datanucleus path> is the location you should put the 3 jar files of livy, and the spark avro jar file if you need to support avro data. - - \<spark conf path> is the location of spark conf directory. ### Build and Run @@ -154,7 +164,7 @@ After a few seconds, we can visit our default UI of Griffin (by default the port http://<your IP>:8080 ``` -You can use UI following the steps [here](../ui/user-guide.md). +You can use UI following the steps [here](../ui/user-guide.md). -**Note**: The front-end UI is still under development, you can only access some basic features currently. +**Note**: The UI doesn't support all the features, for the advanced features you can try API of service. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/dev/dev-env-build.md ---------------------------------------------------------------------- diff --git a/griffin-doc/dev/dev-env-build.md b/griffin-doc/dev/dev-env-build.md index 94e7e8a..2c4cb37 100644 --- a/griffin-doc/dev/dev-env-build.md +++ b/griffin-doc/dev/dev-env-build.md @@ -116,15 +116,15 @@ For end2end test, you will need to build a new griffin docker image, for more co ``` git clone https://github.com/bhlx3lyx7/griffin-docker.git ``` -2. Copy your measure and service JAR into svc_msr_new directory. +2. Copy your measure and service JAR into griffin_spark2 directory. ``` -cp service-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/service/service.jar -cp measure-<version>.jar <path to>/griffin-docker/svc_msr_new/prep/measure/griffin-measure.jar +cp service-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/service/service.jar +cp measure-<version>.jar <path to>/griffin-docker/griffin_spark2/prep/measure/griffin-measure.jar ``` 3. Build your new griffin docker image. -In svc_msr_new directory. +In griffin_spark2 directory. ``` -cd <path to>/griffin-docker/svc_msr_new +cd <path to>/griffin-docker/griffin_spark2 docker build -t <image name>[:<image version>] . ``` 4. If you are using another image name (or version), you need also modify the docker-compose file you're using. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/docker/compose/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/compose/docker-compose-batch.yml b/griffin-doc/docker/compose/docker-compose-batch.yml new file mode 100644 index 0000000..9b247f2 --- /dev/null +++ b/griffin-doc/docker/compose/docker-compose-batch.yml @@ -0,0 +1,45 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/griffin_spark2:0.2.0 + hostname: griffin + links: + - es + environment: + ES_HOSTNAME: es + volumes: + - /var/lib/mysql + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 35432:5432 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + - 39300:9300 + container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/docker/compose/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/compose/docker-compose-streaming.yml b/griffin-doc/docker/compose/docker-compose-streaming.yml new file mode 100644 index 0000000..44759e0 --- /dev/null +++ b/griffin-doc/docker/compose/docker-compose-streaming.yml @@ -0,0 +1,65 @@ +#Licensed to the Apache Software Foundation (ASF) under one +#or more contributor license agreements. See the NOTICE file +#distributed with this work for additional information +#regarding copyright ownership. The ASF licenses this file +#to you under the Apache License, Version 2.0 (the +#"License"); you may not use this file except in compliance +#with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +#Unless required by applicable law or agreed to in writing, +#software distributed under the License is distributed on an +#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +#KIND, either express or implied. See the License for the +#specific language governing permissions and limitations +#under the License. + +griffin: + image: bhlx3lyx7/griffin_spark2:0.2.0 + hostname: griffin + links: + - es + - zk + - kafka + environment: + ES_HOSTNAME: es + ZK_HOSTNAME: zk + KAFKA_HOSTNAME: kafka + volumes: + - /var/lib/mysql + ports: + - 32122:2122 + - 38088:8088 + - 33306:3306 + - 35432:5432 + - 38042:8042 + - 39083:9083 + - 38998:8998 + - 38080:8080 + tty: true + container_name: griffin + +es: + image: bhlx3lyx7/elasticsearch + hostname: es + ports: + - 39200:9200 + - 39300:9300 + container_name: es + +zk: + image: zookeeper:3.5 + hostname: zk + ports: + - 32181:2181 + container_name: zk + restart: always + +kafka: + image: bhlx3lyx7/kafka + hostname: kafka + ports: + - 39092:9092 + container_name: kafka + tty: true \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/docker/griffin-docker-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/griffin-docker-guide.md b/griffin-doc/docker/griffin-docker-guide.md index ff1fae9..9308e7b 100644 --- a/griffin-doc/docker/griffin-docker-guide.md +++ b/griffin-doc/docker/griffin-docker-guide.md @@ -20,44 +20,42 @@ under the License. # Apache Griffin Docker Guide Griffin docker images are pre-built on docker hub, users can pull them to try griffin in docker. -## Environment preparation +## Preparation + +### Environment preparation 1. Install [docker](https://docs.docker.com/engine/installation/) and [docker compose](https://docs.docker.com/compose/install/). 2. Increase vm.max_map_count of your local machine, to use elasticsearch. ``` sysctl -w vm.max_map_count=262144 ``` -## Pull the Docker Images - Here lists the griffin and dependent environment docker images: - - `bhlx3lyx7/svc_msr`: This image contains mysql, hadoop, hive, spark, livy, griffin service, griffin measure, and some prepared demo data, it works as a single node spark cluster, providing spark engine and griffin service. - - `bhlx3lyx7/elasticsearch`: This image is based on official elasticsearch, adding some configurations to enable cors requests, to provide elasticsearch service for metrics persist. - - `bhlx3lyx7/kafka`: This image contains kafka 0.8, and some demo streaming data, to provide streaming data source in streaming mode. - - `zookeeper:3.5`: This image is official zookeeper, to provide zookeeper service in streaming mode. - - Pull the images by the following commands. - ``` - docker pull bhlx3lyx7/svc_msr:0.2.0 +3. Pull griffin pre-built docker images. + ``` + docker pull bhlx3lyx7/griffin_spark2:0.2.0 docker pull bhlx3lyx7/elasticsearch docker pull bhlx3lyx7/kafka docker pull zookeeper:3.5 - ``` + ``` Or you can pull the images faster through mirror acceleration if you are in China. - ``` - docker pull registry.docker-cn.com/bhlx3lyx7/svc_msr:0.2.0 + ``` + docker pull registry.docker-cn.com/bhlx3lyx7/griffin_spark2:0.2.0 docker pull registry.docker-cn.com/bhlx3lyx7/elasticsearch docker pull registry.docker-cn.com/bhlx3lyx7/kafka docker pull registry.docker-cn.com/zookeeper:3.5 - ``` -## Start the Containers -After pulling the images, you can run the docker image in [Batch mode](#how-to-use-griffin-docker-images-in-batch-mode)(always the first choice) or [Streaming mode](#how-to-use-griffin-docker-images-in-streaming-mode)(only if you need a streaming environment like kafka data source). Choose one mode and follow step1&2 to start the docker container. + ``` + The docker images are the griffin environment images. + - `bhlx3lyx7/griffin_spark2`: This image contains mysql, hadoop, hive, spark, livy, griffin service, griffin measure, and some prepared demo data, it works as a single node spark cluster, providing spark engine and griffin service. + - `bhlx3lyx7/elasticsearch`: This image is based on official elasticsearch, adding some configurations to enable cors requests, to provide elasticsearch service for metrics persist. + - `bhlx3lyx7/kafka`: This image contains kafka 0.8, and some demo streaming data, to provide streaming data source in streaming mode. + - `zookeeper:3.5`: This image is official zookeeper, to provide zookeeper service in streaming mode. ### How to use griffin docker images in batch mode -1. Copy [docker-compose-batch.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-batch.yml) to your work path. +1. Copy [docker-compose-batch.yml](compose/docker-compose-batch.yml) to your work path. 2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. ``` docker-compose -f docker-compose-batch.yml up -d ``` -3. Now you can try griffin APIs by using postman after importing the [json files](https://github.com/apache/incubator-griffin/tree/master/griffin-doc/service/postman). - In which you need to modify the environment `BASE_PATH` value into `<your local IP address>:38080`. +3. Now you can try griffin APIs by using postman after importing the [json files](../service/postman). + In which you need to modify the environment `BASE_PATH` value to `<your local IP address>:38080`. 4. You can try the api `Basic -> Get griffin version`, to make sure griffin service has started up. 5. Add an accuracy measure through api `Measures -> Add measure`, to create a measure in griffin. 6. Add a job to through api `jobs -> Add job`, to schedule a job to execute the measure. In the example, the schedule interval is 5 minutes. @@ -67,7 +65,7 @@ After pulling the images, you can run the docker image in [Batch mode](#how-to-u ``` ### How to use griffin docker images in streaming mode -1. Copy [docker-compose-streaming.yml](https://github.com/apache/incubator-griffin/blob/master/griffin-doc/docker/svc_msr/docker-compose-streaming.yml) to your work path. +1. Copy [docker-compose-streaming.yml](compose/docker-compose-streaming.yml) to your work path. 2. In your work path, start docker containers by using docker compose, wait for about one minutes, then griffin service is ready. ``` docker-compose -f docker-compose-streaming.yml up -d @@ -104,4 +102,4 @@ After pulling the images, you can run the docker image in [Batch mode](#how-to-u You can trace the log in streaming-prof.log. ``` tail -f streaming-prof.log - ``` + ``` \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/docker/svc_msr/docker-compose-batch.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-batch.yml b/griffin-doc/docker/svc_msr/docker-compose-batch.yml deleted file mode 100644 index 5375b5c..0000000 --- a/griffin-doc/docker/svc_msr/docker-compose-batch.yml +++ /dev/null @@ -1,45 +0,0 @@ -#Licensed to the Apache Software Foundation (ASF) under one -#or more contributor license agreements. See the NOTICE file -#distributed with this work for additional information -#regarding copyright ownership. The ASF licenses this file -#to you under the Apache License, Version 2.0 (the -#"License"); you may not use this file except in compliance -#with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -#Unless required by applicable law or agreed to in writing, -#software distributed under the License is distributed on an -#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -#KIND, either express or implied. See the License for the -#specific language governing permissions and limitations -#under the License. - -griffin: - image: bhlx3lyx7/svc_msr:0.2.0 - hostname: griffin - links: - - es - environment: - ES_HOSTNAME: es - volumes: - - /var/lib/mysql - ports: - - 32122:2122 - - 38088:8088 - - 33306:3306 - - 35432:5432 - - 38042:8042 - - 39083:9083 - - 38998:8998 - - 38080:8080 - tty: true - container_name: griffin - -es: - image: bhlx3lyx7/elasticsearch - hostname: es - ports: - - 39200:9200 - - 39300:9300 - container_name: es \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/docker/svc_msr/docker-compose-streaming.yml ---------------------------------------------------------------------- diff --git a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml b/griffin-doc/docker/svc_msr/docker-compose-streaming.yml deleted file mode 100644 index bb17f70..0000000 --- a/griffin-doc/docker/svc_msr/docker-compose-streaming.yml +++ /dev/null @@ -1,65 +0,0 @@ -#Licensed to the Apache Software Foundation (ASF) under one -#or more contributor license agreements. See the NOTICE file -#distributed with this work for additional information -#regarding copyright ownership. The ASF licenses this file -#to you under the Apache License, Version 2.0 (the -#"License"); you may not use this file except in compliance -#with the License. You may obtain a copy of the License at -# -# http://www.apache.org/licenses/LICENSE-2.0 -# -#Unless required by applicable law or agreed to in writing, -#software distributed under the License is distributed on an -#"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY -#KIND, either express or implied. See the License for the -#specific language governing permissions and limitations -#under the License. - -griffin: - image: bhlx3lyx7/svc_msr:0.2.0 - hostname: griffin - links: - - es - - zk - - kafka - environment: - ES_HOSTNAME: es - ZK_HOSTNAME: zk - KAFKA_HOSTNAME: kafka - volumes: - - /var/lib/mysql - ports: - - 32122:2122 - - 38088:8088 - - 33306:3306 - - 35432:5432 - - 38042:8042 - - 39083:9083 - - 38998:8998 - - 38080:8080 - tty: true - container_name: griffin - -es: - image: bhlx3lyx7/elasticsearch - hostname: es - ports: - - 39200:9200 - - 39300:9300 - container_name: es - -zk: - image: zookeeper:3.5 - hostname: zk - ports: - - 32181:2181 - container_name: zk - restart: always - -kafka: - image: bhlx3lyx7/kafka - hostname: kafka - ports: - - 39092:9092 - container_name: kafka - tty: true \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/measure/dsl-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/measure/dsl-guide.md b/griffin-doc/measure/dsl-guide.md index 0fc8059..7370274 100644 --- a/griffin-doc/measure/dsl-guide.md +++ b/griffin-doc/measure/dsl-guide.md @@ -125,6 +125,10 @@ Profiling rule expression in Griffin DSL is a sql-like expression, with select c Uniqueness rule expression in Griffin DSL is a list of selection expressions separated by comma, indicates the columns to check if is unique. e.g. `name, age`, `name, (age + 1) as next_age` +### Distinctness Rule +Distinctness rule expression in Griffin DSL is a list of selection expressions separated by comma, indicates the columns to check if is distinct. + e.g. `name, age`, `name, (age + 1) as next_age` + ### Timeliness Rule Timeliness rule expression in Griffin DSL is a list of selection expressions separated by comma, indicates the input time and output time (calculate time as default if not set). e.g. `ts`, `ts, end_ts` @@ -149,12 +153,12 @@ For profiling, the request is always the aggregation function of data, the rule For example, the dsl rule is `source.cntry, source.id.count(), source.age.max() group by source.cntry`, which represents the profiling requests. After the translation, the sql rule is as below: - **profiling sql rule**: `SELECT source.cntry, count(source.id), max(source.age) FROM source GROUP BY source.cntry`, save as table `profiling`. -After the translation, the metrics will be persisted in table `profiling`. +After the translation, the metrics will be persisted in table `profiling`. ### Uniqueness -For uniqueness, or called duplicate, is to find out the duplicate items of data, and rollup the items count group by duplicate times. -For example, the dsl rule is `name, age`, which represents the duplicate requests, in this case, source and target are the same data set. After the translation, the sql rule is as below: -- **get distinct items from source**: `SELECT name, age FROM source`, save as table `src`. +For uniqueness, or called duplicate, is to find out the duplicate items of data, and rollup the items count group by duplicate times. +For example, the dsl rule is `name, age`, which represents the duplicate requests, in this case, source and target are the same data set. After the translation, the sql rule is as below: +- **get distinct items from source**: `SELECT name, age FROM source`, save as table `src`. - **get all items from target**: `SELECT name, age FROM target`, save as table `tgt`. - **join two tables**: `SELECT src.name, src.age FROM tgt RIGHT JOIN src ON coalesce(src.name, '') = coalesce(tgt.name, '') AND coalesce(src.age, '') = coalesce(tgt.age, '')`, save as table `joined`. - **get items duplication**: `SELECT name, age, (count(*) - 1) AS dup FROM joined GROUP BY name, age`, save as table `grouped`. @@ -166,6 +170,20 @@ For example, the dsl rule is `name, age`, which represents the duplicate request After the translation, the metrics will be persisted in table `dup_metric`. +### Distinctness +For distinctness, is to find out the duplicate items of data, the same as uniqueness in batch mode, but with some differences in streaming mode. +In most time, you need distinctness other than uniqueness. +For example, the dsl rule is `name, age`, which represents the distinct requests, in this case, source and target are the same data set. After the translation, the sql rule is as below: +- **total count of source**: `SELECT COUNT(*) AS total FROM source`, save as table `total_count`. +- **group by fields**: `SELECT name, age, (COUNT(*) - 1) AS dup, TRUE AS dist FROM source GROUP BY name, age`, save as table `dup_count`. +- **distinct metric**: `SELECT COUNT(*) AS dist_count FROM dup_count WHERE dist`, save as table `distinct_metric`. +- **source join distinct metric**: `SELECT source.*, dup_count.dup AS dup, dup_count.dist AS dist FROM source LEFT JOIN dup_count ON source.name = dup_count.name AND source.age = dup_count.age`, save as table `dist_joined`. +- **add row number**: `SELECT *, ROW_NUMBER() OVER (DISTRIBUTE BY name, age SORT BY dist) row_num FROM dist_joined`, save as table `row_numbered`. +- **duplicate records**: `SELECT name, age, dup FROM row_numbered WHERE NOT dist OR row_num > 1`, save as table `dup_records`. +- **duplicate metric**: `SELECT name, age, dup, COUNT(*) AS num FROM dup_records GROUP BY name, age, dup`, save as table `dup_metric`. + +After the translation, the metrics will be persisted in table `distinct_metric` and `dup_metric`. + ### Timeliness For timeliness, is to measure the latency of each item, and get the statistics of the latencies. For example, the dsl rule is `ts, out_ts`, the first column means the input time of item, the second column means the output time of item, if not set, `__tmst` will be the default output time column. After the translation, the sql rule is as below: http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/measure/measure-configuration-guide.md ---------------------------------------------------------------------- diff --git a/griffin-doc/measure/measure-configuration-guide.md b/griffin-doc/measure/measure-configuration-guide.md index 0db3643..53f173c 100644 --- a/griffin-doc/measure/measure-configuration-guide.md +++ b/griffin-doc/measure/measure-configuration-guide.md @@ -191,8 +191,8 @@ Above lists DQ job configure parameters. + hive data connector * database: data base name, optional, "default" as default. * table.name: table name. - * partitions: partition conditions string, split by ";" and ",", optional. - e.g. `dt=20170410, hour=15; dt=20170411, hour=15; dt=20170412, hour=15` + * where: where conditions string, split by ",", optional. + e.g. `dt=20170410 AND hour=15, dt=20170411 AND hour=15, dt=20170412 AND hour=15` + text dir data connector * dir.path: parent directory path. * data.dir.depth: integer, depth of data directories, 0 as default. http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/griffin-doc/roadmap.md ---------------------------------------------------------------------- diff --git a/griffin-doc/roadmap.md b/griffin-doc/roadmap.md index b916311..01807e8 100644 --- a/griffin-doc/roadmap.md +++ b/griffin-doc/roadmap.md @@ -21,41 +21,32 @@ under the License. ## Current feature list In the current version, we've implemented the below main DQ features -- **Data Asset Management** - - User can register, delete, edit data assets, currently only Hadoop data-sets are supported +- **Data Asset Detection** + After configuration in service module, Griffin can detect the Hive tables metadata through Hive metastore service. - **Measure Management** + Through UI, user can create, delete measures for 3 types: accuracy, profiling and publish metrics. + Through service API, user can create, delete and update measures for 6 types: accuracy, profiling, timeliness, uniqueness, completeness and publish metrics. - User can create, delete, edit measures for 4 types: Accuracy, Profiling, Anomaly Detection, Publish Metrics - -- **Job Scheduler** - - After the measures are created, the Job Scheduler component can create the jobs and schedule them to calculate the metrics values +- **Job Management** + User can create, delete job to schedule batch job for calculative measures, data range of each calculation, and the extra trigger condition like "done file" on hdfs. -- **Measure Execution on Spark** - - The Job Scheduler will trigger the measure execution on Spark to generate the metrics values +- **Measure Calculation on Spark** + Service module will trigger and submit calculation jobs to Spark cluster through livy, the measure module calculates and persists the metric values to elasticsearch by default. - **Metrics Visualization** - - We have a web portal to display all metrics - -- **My Dashboard** - - Only the interested metrics will be displayed on "My Dashboard" + Through service API, user can get metric values of each job from elasticsearch. + On UI, accuracy metrics will be rendered as a chart, profiling metrics will be displayed as a table. ## Short-term Roadmap -- **Support more data-set types** - - Current we only support Hadoop datasets, we should also support RDBMS and real-time streaming data from Kafka, Storm, etc. +- **Support more data source types** + At current, Griffin only supports Hive table, avro files on hdfs as data source in batch mode, Kafka as data source in streaming mode. + We plan to support more data source types, like RDBM, elasticsearch. - **Support more data quality dimensions** + Griffin need to support more data quality dimensions, like consistency and validity. - Besides accuracy, there are some other data quality dimensions(Completeness, Uniqueness, Timeliness, Validity, Consistency), we should support more dimensions - -- **More ML algorithms for Anomaly Detection measure** - - Currently only [MAD(Median absolute deviation)](https://en.wikipedia.org/wiki/Median_absolute_deviation) and [Bollinger Bands](https://en.wikipedia.org/wiki/Bollinger_Bands) are supported, we are considering to support more Machine Learning algorithms +- **Anomaly Detection** + Griffin plan to support anomaly detection, by analyzing calculated metrics from elasticsearch. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/licenses/BSD-2-Clause ---------------------------------------------------------------------- diff --git a/licenses/BSD-2-Clause b/licenses/BSD-2-Clause deleted file mode 100644 index c964b53..0000000 --- a/licenses/BSD-2-Clause +++ /dev/null @@ -1,22 +0,0 @@ -The BSD 2-Clause License - - Redistribution and use in source and binary forms, with or without - modification, are permitted provided that the following conditions are met: - - 1. Redistributions of source code must retain the above copyright notice, this - list of conditions and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright notice, - this list of conditions and the following disclaimer in the documentation - and/or other materials provided with the distribution. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE - DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE - FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL - DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR - SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER - CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, - OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE - OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/licenses/BSD-3-Clause ---------------------------------------------------------------------- diff --git a/licenses/BSD-3-Clause b/licenses/BSD-3-Clause deleted file mode 100644 index c3ca1b9..0000000 --- a/licenses/BSD-3-Clause +++ /dev/null @@ -1,26 +0,0 @@ -The BSD 3-Clause License ("New BSD") - - Redistribution and use in source and binary forms, with or without modification, - are permitted provided that the following conditions are met: - - 1. Redistributions of source code must retain the above copyright notice, - this list of conditions and the following disclaimer. - - 2. Redistributions in binary form must reproduce the above copyright notice, - this list of conditions and the following disclaimer in the documentation - and/or other materials provided with the distribution. - - 3. Neither the name of the copyright holder nor the names of its contributors - may be used to endorse or promote products derived from this software without - specific prior written permission. - - THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND - ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED - WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. - IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, - INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT - NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR - PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, - WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) - ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE - POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-griffin/blob/47ae1774/licenses/CDDL-1.0 ---------------------------------------------------------------------- diff --git a/licenses/CDDL-1.0 b/licenses/CDDL-1.0 deleted file mode 100644 index 253f4cc..0000000 --- a/licenses/CDDL-1.0 +++ /dev/null @@ -1,363 +0,0 @@ -Common Development and Distribution License 1.0 - - 1. Definitions. - - 1.1. "Contributor" means each individual or entity that creates - or contributes to the creation of Modifications. - - 1.2. "Contributor Version" means the combination of the Original - Software, prior Modifications used by a Contributor (if any), - and the Modifications made by that particular Contributor. - - 1.3. "Covered Software" means (a) the Original Software, or (b) - Modifications, or (c) the combination of files containing - Original Software with files containing Modifications, in - each case including portions thereof. - - 1.4. "Executable" means the Covered Software in any form other - than Source Code. - - 1.5. "Initial Developer" means the individual or entity that first - makes Original Software available under this License. - - 1.6. "Larger Work" means a work which combines Covered Software or - portions thereof with code not governed by the terms of this - License. - - 1.7. "License" means this document. - - 1.8. "Licensable" means having the right to grant, to the maximum - extent possible, whether at the time of the initial grant or - subsequently acquired, any and all of the rights conveyed - herein. - - 1.9. "Modifications" means the Source Code and Executable form of - any of the following: - - A. Any file that results from an addition to, deletion from or - modification of the contents of a file containing Original - Software or previous Modifications; - - B. Any new file that contains any part of the Original - Software or previous Modifications; or - - C. Any new file that is contributed or otherwise made - available under the terms of this License. - - 1.10. "Original Software" means the Source Code and Executable - form of computer software code that is originally released - under this License. - - 1.11. "Patent Claims" means any patent claim(s), now owned or - hereafter acquired, including without limitation, method, - process, and apparatus claims, in any patent Licensable by - grantor. - - 1.12. "Source Code" means (a) the common form of computer software - code in which modifications are made and (b) associated - documentation included in or with such code. - - 1.13. "You" (or "Your") means an individual or a legal entity - exercising rights under, and complying with all of the terms - of, this License. For legal entities, "You" includes any - entity which controls, is controlled by, or is under common - control with You. For purposes of this definition, - "control" means (a) the power, direct or indirect, to cause - the direction or management of such entity, whether by - contract or otherwise, or (b) ownership of more than fifty - percent (50%) of the outstanding shares or beneficial - ownership of such entity. - - 2. License Grants. - - 2.1. The Initial Developer Grant. - - Conditioned upon Your compliance with Section 3.1 below and - subject to third party intellectual property claims, the Initial - Developer hereby grants You a world-wide, royalty-free, - non-exclusive license: - - (a) under intellectual property rights (other than patent or - trademark) Licensable by Initial Developer, to use, - reproduce, modify, display, perform, sublicense and - distribute the Original Software (or portions thereof), - with or without Modifications, and/or as part of a Larger - Work; and - - (b) under Patent Claims infringed by the making, using or - selling of Original Software, to make, have made, use, - practice, sell, and offer for sale, and/or otherwise - dispose of the Original Software (or portions thereof). - - (c) The licenses granted in Sections 2.1(a) and (b) are - effective on the date Initial Developer first distributes - or otherwise makes the Original Software available to a - third party under the terms of this License. - - (d) Notwithstanding Section 2.1(b) above, no patent license is - granted: (1) for code that You delete from the Original - Software, or (2) for infringements caused by: (i) the - modification of the Original Software, or (ii) the - combination of the Original Software with other software - or devices. - - 2.2. Contributor Grant. - - Conditioned upon Your compliance with Section 3.1 below and - subject to third party intellectual property claims, each - Contributor hereby grants You a world-wide, royalty-free, - non-exclusive license: - - (a) under intellectual property rights (other than patent or - trademark) Licensable by Contributor to use, reproduce, - modify, display, perform, sublicense and distribute the - Modifications created by such Contributor (or portions - thereof), either on an unmodified basis, with other - Modifications, as Covered Software and/or as part of a - Larger Work; and - - (b) under Patent Claims infringed by the making, using, or - selling of Modifications made by that Contributor either - alone and/or in combination with its Contributor Version - (or portions of such combination), to make, use, sell, - offer for sale, have made, and/or otherwise dispose of: - (1) Modifications made by that Contributor (or portions - thereof); and (2) the combination of Modifications made by - that Contributor with its Contributor Version (or portions - of such combination). - - (c) The licenses granted in Sections 2.2(a) and 2.2(b) are - effective on the date Contributor first distributes or - otherwise makes the Modifications available to a third - party. - - (d) Notwithstanding Section 2.2(b) above, no patent license is - granted: (1) for any code that Contributor has deleted - from the Contributor Version; (2) for infringements caused - by: (i) third party modifications of Contributor Version, - or (ii) the combination of Modifications made by that - Contributor with other software (except as part of the - Contributor Version) or other devices; or (3) under Patent - Claims infringed by Covered Software in the absence of - Modifications made by that Contributor. - - 3. Distribution Obligations. - - 3.1. Availability of Source Code. - - Any Covered Software that You distribute or otherwise make - available in Executable form must also be made available in Source - Code form and that Source Code form must be distributed only under - the terms of this License. You must include a copy of this - License with every copy of the Source Code form of the Covered - Software You distribute or otherwise make available. You must - inform recipients of any such Covered Software in Executable form - as to how they can obtain such Covered Software in Source Code - form in a reasonable manner on or through a medium customarily - used for software exchange. - - 3.2. Modifications. - - The Modifications that You create or to which You contribute are - governed by the terms of this License. You represent that You - believe Your Modifications are Your original creation(s) and/or - You have sufficient rights to grant the rights conveyed by this - License. - - 3.3. Required Notices. - - You must include a notice in each of Your Modifications that - identifies You as the Contributor of the Modification. You may - not remove or alter any copyright, patent or trademark notices - contained within the Covered Software, or any notices of licensing - or any descriptive text giving attribution to any Contributor or - the Initial Developer. - - 3.4. Application of Additional Terms. - - You may not offer or impose any terms on any Covered Software in - Source Code form that alters or restricts the applicable version - of this License or the recipients' rights hereunder. You may - choose to offer, and to charge a fee for, warranty, support, - indemnity or liability obligations to one or more recipients of - Covered Software. However, you may do so only on Your own behalf, - and not on behalf of the Initial Developer or any Contributor. - You must make it absolutely clear that any such warranty, support, - indemnity or liability obligation is offered by You alone, and You - hereby agree to indemnify the Initial Developer and every - Contributor for any liability incurred by the Initial Developer or - such Contributor as a result of warranty, support, indemnity or - liability terms You offer. - - 3.5. Distribution of Executable Versions. - - You may distribute the Executable form of the Covered Software - under the terms of this License or under the terms of a license of - Your choice, which may contain terms different from this License, - provided that You are in compliance with the terms of this License - and that the license for the Executable form does not attempt to - limit or alter the recipient's rights in the Source Code form from - the rights set forth in this License. If You distribute the - Covered Software in Executable form under a different license, You - must make it absolutely clear that any terms which differ from - this License are offered by You alone, not by the Initial - Developer or Contributor. You hereby agree to indemnify the - Initial Developer and every Contributor for any liability incurred - by the Initial Developer or such Contributor as a result of any - such terms You offer. - - 3.6. Larger Works. - - You may create a Larger Work by combining Covered Software with - other code not governed by the terms of this License and - distribute the Larger Work as a single product. In such a case, - You must make sure the requirements of this License are fulfilled - for the Covered Software. - - 4. Versions of the License. - - 4.1. New Versions. - - Sun Microsystems, Inc. is the initial license steward and may - publish revised and/or new versions of this License from time to - time. Each version will be given a distinguishing version number. - Except as provided in Section 4.3, no one other than the license - steward has the right to modify this License. - - 4.2. Effect of New Versions. - - You may always continue to use, distribute or otherwise make the - Covered Software available under the terms of the version of the - License under which You originally received the Covered Software. - If the Initial Developer includes a notice in the Original - Software prohibiting it from being distributed or otherwise made - available under any subsequent version of the License, You must - distribute and make the Covered Software available under the terms - of the version of the License under which You originally received - the Covered Software. Otherwise, You may also choose to use, - distribute or otherwise make the Covered Software available under - the terms of any subsequent version of the License published by - the license steward. - - 4.3. Modified Versions. - - When You are an Initial Developer and You want to create a new - license for Your Original Software, You may create and use a - modified version of this License if You: (a) rename the license - and remove any references to the name of the license steward - (except to note that the license differs from this License); and - (b) otherwise make it clear that the license contains terms which - differ from this License. - - 5. DISCLAIMER OF WARRANTY. - - COVERED SOFTWARE IS PROVIDED UNDER THIS LICENSE ON AN "AS IS" - BASIS, WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, - INCLUDING, WITHOUT LIMITATION, WARRANTIES THAT THE COVERED - SOFTWARE IS FREE OF DEFECTS, MERCHANTABLE, FIT FOR A PARTICULAR - PURPOSE OR NON-INFRINGING. THE ENTIRE RISK AS TO THE QUALITY AND - PERFORMANCE OF THE COVERED SOFTWARE IS WITH YOU. SHOULD ANY - COVERED SOFTWARE PROVE DEFECTIVE IN ANY RESPECT, YOU (NOT THE - INITIAL DEVELOPER OR ANY OTHER CONTRIBUTOR) ASSUME THE COST OF ANY - NECESSARY SERVICING, REPAIR OR CORRECTION. THIS DISCLAIMER OF - WARRANTY CONSTITUTES AN ESSENTIAL PART OF THIS LICENSE. NO USE OF - ANY COVERED SOFTWARE IS AUTHORIZED HEREUNDER EXCEPT UNDER THIS - DISCLAIMER. - - 6. TERMINATION. - - 6.1. This License and the rights granted hereunder will terminate - automatically if You fail to comply with terms herein and fail to - cure such breach within 30 days of becoming aware of the breach. - Provisions which, by their nature, must remain in effect beyond - the termination of this License shall survive. - - 6.2. If You assert a patent infringement claim (excluding - declaratory judgment actions) against Initial Developer or a - Contributor (the Initial Developer or Contributor against whom You - assert such claim is referred to as "Participant") alleging that - the Participant Software (meaning the Contributor Version where - the Participant is a Contributor or the Original Software where - the Participant is the Initial Developer) directly or indirectly - infringes any patent, then any and all rights granted directly or - indirectly to You by such Participant, the Initial Developer (if - the Initial Developer is not the Participant) and all Contributors - under Sections 2.1 and/or 2.2 of this License shall, upon 60 days - notice from Participant terminate prospectively and automatically - at the expiration of such 60 day notice period, unless if within - such 60 day period You withdraw Your claim with respect to the - Participant Software against such Participant either unilaterally - or pursuant to a written agreement with Participant. - - 6.3. In the event of termination under Sections 6.1 or 6.2 above, - all end user licenses that have been validly granted by You or any - distributor hereunder prior to termination (excluding licenses - granted to You by any distributor) shall survive termination. - - 7. LIMITATION OF LIABILITY. - - UNDER NO CIRCUMSTANCES AND UNDER NO LEGAL THEORY, WHETHER TORT - (INCLUDING NEGLIGENCE), CONTRACT, OR OTHERWISE, SHALL YOU, THE - INITIAL DEVELOPER, ANY OTHER CONTRIBUTOR, OR ANY DISTRIBUTOR OF - COVERED SOFTWARE, OR ANY SUPPLIER OF ANY OF SUCH PARTIES, BE - LIABLE TO ANY PERSON FOR ANY INDIRECT, SPECIAL, INCIDENTAL, OR - CONSEQUENTIAL DAMAGES OF ANY CHARACTER INCLUDING, WITHOUT - LIMITATION, DAMAGES FOR LOST PROFITS, LOSS OF GOODWILL, WORK - STOPPAGE, COMPUTER FAILURE OR MALFUNCTION, OR ANY AND ALL OTHER - COMMERCIAL DAMAGES OR LOSSES, EVEN IF SUCH PARTY SHALL HAVE BEEN - INFORMED OF THE POSSIBILITY OF SUCH DAMAGES. THIS LIMITATION OF - LIABILITY SHALL NOT APPLY TO LIABILITY FOR DEATH OR PERSONAL - INJURY RESULTING FROM SUCH PARTY'S NEGLIGENCE TO THE EXTENT - APPLICABLE LAW PROHIBITS SUCH LIMITATION. SOME JURISDICTIONS DO - NOT ALLOW THE EXCLUSION OR LIMITATION OF INCIDENTAL OR - CONSEQUENTIAL DAMAGES, SO THIS EXCLUSION AND LIMITATION MAY NOT - APPLY TO YOU. - - 8. U.S. GOVERNMENT END USERS. - - The Covered Software is a "commercial item," as that term is - defined in 48 C.F.R. 2.101 (Oct. 1995), consisting of "commercial - computer software" (as that term is defined at 48 - C.F.R. 252.227-7014(a)(1)) and "commercial computer software - documentation" as such terms are used in 48 C.F.R. 12.212 - (Sept. 1995). Consistent with 48 C.F.R. 12.212 and 48 - C.F.R. 227.7202-1 through 227.7202-4 (June 1995), all - U.S. Government End Users acquire Covered Software with only those - rights set forth herein. This U.S. Government Rights clause is in - lieu of, and supersedes, any other FAR, DFAR, or other clause or - provision that addresses Government rights in computer software - under this License. - - 9. MISCELLANEOUS. - - This License represents the complete agreement concerning subject - matter hereof. If any provision of this License is held to be - unenforceable, such provision shall be reformed only to the extent - necessary to make it enforceable. This License shall be governed - by the law of the jurisdiction specified in a notice contained - within the Original Software (except to the extent applicable law, - if any, provides otherwise), excluding such jurisdiction's - conflict-of-law provisions. Any litigation relating to this - License shall be subject to the jurisdiction of the courts located - in the jurisdiction and venue specified in a notice contained - within the Original Software, with the losing party responsible - for costs, including, without limitation, court costs and - reasonable attorneys' fees and expenses. The application of the - United Nations Convention on Contracts for the International Sale - of Goods is expressly excluded. Any law or regulation which - provides that the language of a contract shall be construed - against the drafter shall not apply to this License. You agree - that You alone are responsible for compliance with the United - States export administration regulations (and the export control - laws and regulation of any other countries) when You use, - distribute or otherwise make available any Covered Software. - - 10. RESPONSIBILITY FOR CLAIMS. - - As between Initial Developer and the Contributors, each party is - responsible for claims and damages arising, directly or - indirectly, out of its utilization of rights under this License - and You agree to work with Initial Developer and Contributors to - distribute such responsibility on an equitable basis. Nothing - herein is intended or shall be deemed to constitute any admission - of liability. \ No newline at end of file
